Oct 31 07:26:32.969208 ip-10-0-140-6 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 31 07:26:32.969222 ip-10-0-140-6 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 31 07:26:32.969231 ip-10-0-140-6 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 31 07:26:32.969562 ip-10-0-140-6 systemd[1]: Failed to start Kubernetes Kubelet. Oct 31 07:26:43.218135 ip-10-0-140-6 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 31 07:26:43.218151 ip-10-0-140-6 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 3a9a386413414820b8e4fbdea582fc0a -- Oct 31 07:28:34.285082 ip-10-0-140-6 systemd[1]: Starting Kubernetes Kubelet... Oct 31 07:28:34.717591 ip-10-0-140-6 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 07:28:34.717591 ip-10-0-140-6 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 07:28:34.717591 ip-10-0-140-6 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 07:28:34.717591 ip-10-0-140-6 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 07:28:34.721143 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721068 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.721143 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721142 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.721143 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721146 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721151 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721154 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721157 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721161 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721164 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721166 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721169 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721172 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721175 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721185 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721188 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721192 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721194 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721197 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721199 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721202 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721205 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721207 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721210 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.721241 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721212 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721215 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721217 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721221 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721223 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721226 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721228 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721231 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721233 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721236 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721238 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721241 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721244 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721246 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721249 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721253 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721255 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721258 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721262 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721266 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.721746 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721268 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721271 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721274 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721276 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721280 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721282 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721285 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721287 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721290 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721293 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721296 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721298 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721301 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721303 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721307 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721310 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721312 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721315 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721317 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721320 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.722218 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721323 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721325 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721328 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721330 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721333 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721336 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721338 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721341 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721343 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721346 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721348 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721351 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721353 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721356 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721358 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721362 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721368 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721371 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721375 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.722716 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721378 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721381 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721383 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721386 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721389 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721391 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721393 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721397 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721399 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721402 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721405 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721407 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721410 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721412 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721415 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721417 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721420 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721423 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721425 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.723454 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721427 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721430 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721432 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721435 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721438 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.721440 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722201 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722208 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722211 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722214 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722217 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722221 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722224 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722226 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722229 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722232 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722235 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722239 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722243 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.724179 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722247 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722251 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722253 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722256 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722259 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722261 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722264 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722267 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722270 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722273 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722275 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722278 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722280 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722283 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722285 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722288 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722290 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722293 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722295 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722298 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.724877 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722300 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722303 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722305 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722307 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722310 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722313 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722316 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722318 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722321 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722323 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722326 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722329 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722331 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722335 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722338 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722340 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722343 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722345 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722348 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722350 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.725351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722353 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722355 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722358 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722360 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722363 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722366 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722368 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722372 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722374 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722378 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722381 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722383 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722386 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722389 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722391 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722394 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722396 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722399 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722402 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722404 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.726036 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722407 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722410 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722412 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722415 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722417 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722421 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722423 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722426 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722429 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722431 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722434 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722436 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722439 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722441 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722443 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722446 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722448 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722451 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722453 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722456 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.726905 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722459 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722461 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722464 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722466 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722468 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722471 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722473 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722476 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722478 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722481 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722486 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722490 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.722493 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723746 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723764 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723771 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723775 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723781 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723786 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.727754 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723807 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723812 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723816 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723821 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723825 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723829 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723839 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723844 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723848 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723852 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723856 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723860 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723864 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723868 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723873 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723877 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723881 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723885 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723889 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.728353 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723898 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723902 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723907 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723911 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723915 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723919 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723923 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723927 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723933 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723937 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723941 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723945 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723950 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723960 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723964 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723967 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723971 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723975 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723980 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723984 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.728903 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723987 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723991 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723995 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.723999 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724004 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724012 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724016 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724020 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724025 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724029 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724033 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724038 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724050 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724057 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724091 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724242 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724252 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724258 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724263 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.729524 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724271 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724281 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724287 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724292 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724297 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724301 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724306 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724310 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724314 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724318 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724323 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724327 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724332 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724336 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724340 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724344 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724349 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724353 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724358 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724362 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.729990 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724366 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724371 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724375 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724379 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724383 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724387 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724392 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724398 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724402 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724407 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724412 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724417 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724421 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724425 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724429 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724434 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724438 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724442 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724449 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.730464 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724459 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724463 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.724467 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725679 2580 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725696 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725708 2580 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725715 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725723 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725728 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725735 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725741 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725746 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725751 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725757 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725763 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725769 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725774 2580 flags.go:64] FLAG: --cgroup-root="" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725779 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725784 2580 flags.go:64] FLAG: --client-ca-file="" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725806 2580 flags.go:64] FLAG: --cloud-provider="external" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725813 2580 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725821 2580 flags.go:64] FLAG: --cluster-domain="" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725826 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725831 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 07:28:34.731017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725837 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725842 2580 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725848 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725853 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725859 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725864 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725869 2580 flags.go:64] FLAG: --contention-profiling="false" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725873 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725878 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725884 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725889 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725896 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725901 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725906 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725910 2580 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725915 2580 flags.go:64] FLAG: --enable-server="true" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725921 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725928 2580 flags.go:64] FLAG: --event-burst="100" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725934 2580 flags.go:64] FLAG: --event-qps="50" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725939 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725944 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725949 2580 flags.go:64] FLAG: --eviction-hard="" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725956 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725961 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725966 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 07:28:34.731580 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725971 2580 flags.go:64] FLAG: --eviction-soft="" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725977 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725982 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725987 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725991 2580 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.725996 2580 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726001 2580 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726006 2580 flags.go:64] FLAG: --feature-gates="" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726012 2580 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726017 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726022 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726029 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726034 2580 flags.go:64] FLAG: --healthz-port="10248" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726039 2580 flags.go:64] FLAG: --help="false" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726043 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-6.ec2.internal" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726048 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726053 2580 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726058 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726063 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726070 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726075 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726080 2580 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726084 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 07:28:34.732265 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726089 2580 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726094 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726099 2580 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726103 2580 flags.go:64] FLAG: --kube-reserved="" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726108 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726113 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726118 2580 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726125 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726130 2580 flags.go:64] FLAG: --lock-file="" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726134 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726139 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726144 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726154 2580 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726159 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726164 2580 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726168 2580 flags.go:64] FLAG: --logging-format="text" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726173 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726178 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726183 2580 flags.go:64] FLAG: --manifest-url="" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726188 2580 flags.go:64] FLAG: --manifest-url-header="" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726195 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726200 2580 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726206 2580 flags.go:64] FLAG: --max-pods="110" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726211 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726216 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 07:28:34.732846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726221 2580 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726226 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726230 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726235 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726241 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726252 2580 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726257 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726262 2580 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726268 2580 flags.go:64] FLAG: --pod-cidr="" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726272 2580 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726277 2580 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726282 2580 flags.go:64] FLAG: --pods-per-core="0" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726287 2580 flags.go:64] FLAG: --port="10250" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726291 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726296 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0564f5668d95865ef" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726302 2580 flags.go:64] FLAG: --qos-reserved="" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726307 2580 flags.go:64] FLAG: --read-only-port="10255" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726312 2580 flags.go:64] FLAG: --register-node="true" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726317 2580 flags.go:64] FLAG: --register-with-taints="" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726331 2580 flags.go:64] FLAG: --registry-burst="10" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726336 2580 flags.go:64] FLAG: --registry-qps="5" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726341 2580 flags.go:64] FLAG: --reserved-cpus="" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726345 2580 flags.go:64] FLAG: --reserved-memory="" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726351 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726356 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726361 2580 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 07:28:34.733435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726366 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726371 2580 flags.go:64] FLAG: --runonce="false" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726375 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726380 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726385 2580 flags.go:64] FLAG: --seccomp-default="false" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726390 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726395 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726400 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726405 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726410 2580 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726415 2580 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726420 2580 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726424 2580 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726429 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726433 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726439 2580 flags.go:64] FLAG: --system-cgroups="" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726444 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726454 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726459 2580 flags.go:64] FLAG: --tls-cert-file="" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726464 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726471 2580 flags.go:64] FLAG: --tls-min-version="" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726476 2580 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726482 2580 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726487 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726491 2580 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 07:28:34.734110 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726496 2580 flags.go:64] FLAG: --v="2" Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726503 2580 flags.go:64] FLAG: --version="false" Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726510 2580 flags.go:64] FLAG: --vmodule="" Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726516 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.726522 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726770 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726777 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726782 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726787 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726810 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726823 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726828 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726833 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726837 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726842 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726854 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726862 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726867 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726872 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726877 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.734692 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726882 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726886 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726891 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726895 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726899 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726903 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726907 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726912 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726915 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726919 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726925 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726929 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726934 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726939 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726944 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726948 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726954 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726961 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726966 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.735183 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726970 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726975 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726979 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726984 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726988 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726992 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.726997 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727001 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727005 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727014 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727019 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727024 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727028 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727032 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727036 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727044 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727048 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727053 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727057 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.735619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727062 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727066 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727071 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727075 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727079 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727084 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727091 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727097 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727101 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727106 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727110 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727115 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727119 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727123 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727128 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727132 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727136 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727141 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727145 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.736109 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727150 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727154 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727158 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727164 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727169 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727173 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727177 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727181 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727186 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727191 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727196 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727200 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727204 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727209 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727213 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727218 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727222 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727226 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727231 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727236 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.736552 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727241 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727245 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727249 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727254 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727258 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727262 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727266 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727270 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727275 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727279 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727283 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727288 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727292 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727296 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.727341 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727408 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.737126 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727414 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727418 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727423 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727428 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727432 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727436 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727442 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727446 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727451 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727455 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727459 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727463 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727468 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727472 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727476 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727483 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727491 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727496 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727501 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.737494 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727506 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727510 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727514 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727519 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727523 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727527 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727531 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727536 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727540 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727544 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727548 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727553 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727558 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727563 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727568 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727572 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727576 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727580 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727585 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727590 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.737946 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727595 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727599 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727604 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727608 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727612 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727617 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727621 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727625 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727630 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727634 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727639 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727644 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727648 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727652 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727656 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727661 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727665 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727669 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727673 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727677 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.738414 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727681 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727686 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727690 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727694 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727700 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727704 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727709 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727713 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727717 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727721 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727726 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727731 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727736 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727740 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727744 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727748 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727752 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727758 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727762 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727766 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.738910 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727770 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727774 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727778 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727783 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727803 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727808 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727812 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727817 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727821 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727825 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727830 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727834 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727839 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727843 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727849 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727856 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727862 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727866 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727871 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.739370 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727875 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727879 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727884 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727888 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727896 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727900 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.727905 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.735568 2580 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.735585 2580 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.735602 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.735607 2580 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735662 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735667 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735670 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735673 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735676 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735679 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735681 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.739813 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735686 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735690 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735693 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735696 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735699 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735702 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735705 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735707 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735710 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735713 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735716 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735718 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735721 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735724 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735727 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735729 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735732 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735734 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735737 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735740 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.740237 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735743 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735746 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735749 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735752 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735755 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735758 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735760 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735763 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735766 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735768 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735771 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735773 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735776 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735778 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735782 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735786 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735807 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735810 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735814 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735817 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.740689 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735819 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735822 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735825 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735827 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735830 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735832 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735835 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735838 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735841 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735844 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735847 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735850 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735852 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735855 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735858 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735861 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735864 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735867 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735869 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735872 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.741258 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735875 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735878 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735880 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735882 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735885 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735887 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735890 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735892 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735895 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735898 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735900 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735903 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735906 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735909 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735911 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735913 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735916 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735919 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735922 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735924 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.741751 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735927 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735930 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735932 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735935 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735938 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735941 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735944 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735946 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735949 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735952 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735955 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735957 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735960 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735963 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735965 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735968 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735971 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735973 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.742245 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.735976 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.736004 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736143 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736148 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736151 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736154 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736157 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736159 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736162 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736165 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736168 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736170 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736173 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736176 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736178 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736181 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.742652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736184 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736186 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736189 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736192 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736195 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736197 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736200 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736203 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736206 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736208 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736211 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736214 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736216 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736219 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736221 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736224 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736227 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736229 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736232 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736234 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.743050 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736237 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736239 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736242 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736245 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736247 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736250 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736252 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736255 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736257 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736260 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736262 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736265 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736267 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736270 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736273 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736276 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736278 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736281 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736283 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736286 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.743540 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736289 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736292 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736294 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736297 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736299 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736302 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736304 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736307 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736309 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736312 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736316 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736320 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736322 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736325 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736327 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736330 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736332 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736335 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.744022 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736337 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736340 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736342 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736345 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736347 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736350 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736352 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736355 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736357 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736360 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736363 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736365 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736367 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736371 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736373 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736376 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736379 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736382 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736387 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.744440 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736389 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736392 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736395 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736398 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736400 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736403 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736405 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736408 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736410 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736413 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736415 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736418 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736420 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736423 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736426 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.736448 2580 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 07:28:34.744899 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736479 2580 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736483 2580 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736486 2580 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736489 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736491 2580 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736494 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736497 2580 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736499 2580 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736502 2580 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736505 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736507 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736510 2580 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736512 2580 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736515 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736518 2580 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736521 2580 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736523 2580 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736526 2580 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736529 2580 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736531 2580 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:28:34.745282 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736534 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736536 2580 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736539 2580 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736541 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736544 2580 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736546 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736549 2580 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736551 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736554 2580 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736556 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736559 2580 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736562 2580 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736565 2580 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736568 2580 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736572 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736575 2580 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736577 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736580 2580 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:28:34.745742 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736583 2580 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736585 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736588 2580 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736590 2580 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736593 2580 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736595 2580 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736598 2580 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736601 2580 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736604 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736606 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736609 2580 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736611 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736614 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736616 2580 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736619 2580 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736621 2580 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736624 2580 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736626 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736629 2580 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736631 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:28:34.746180 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736634 2580 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736637 2580 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736642 2580 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736645 2580 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736647 2580 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736650 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736652 2580 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736655 2580 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736657 2580 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736660 2580 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736662 2580 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736665 2580 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736667 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736670 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736672 2580 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736675 2580 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736677 2580 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736680 2580 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:28:34.746636 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736682 2580 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736685 2580 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736687 2580 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736691 2580 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736693 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736696 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736698 2580 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736701 2580 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736704 2580 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736706 2580 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736709 2580 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736711 2580 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736714 2580 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736716 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736722 2580 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736724 2580 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736727 2580 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736729 2580 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736732 2580 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736734 2580 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736737 2580 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:28:34.747068 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736739 2580 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736742 2580 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736745 2580 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736747 2580 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736751 2580 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736753 2580 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736756 2580 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736758 2580 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:34.736761 2580 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.736834 2580 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.739182 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.740259 2580 server.go:1014] "Starting client certificate rotation" Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.740350 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.740374 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 07:28:34.747537 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.742547 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 07:28:34.747892 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.747652 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 07:28:34.760922 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.760901 2580 log.go:25] "Validated CRI v1 runtime API" Oct 31 07:28:34.764727 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.764712 2580 log.go:25] "Validated CRI v1 image API" Oct 31 07:28:34.765548 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.765522 2580 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 07:28:34.768605 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.768551 2580 fs.go:135] Filesystem UUIDs: map[17aef96a-daa3-461d-8a39-7c96daa5ad65:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dab11254-8006-4d6d-bead-2949b25410f8:/dev/nvme0n1p4] Oct 31 07:28:34.768704 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.768605 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 07:28:34.774161 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.774045 2580 manager.go:217] Machine: {Timestamp:2026-10-31 07:28:34.77255912 +0000 UTC m=+0.376404696 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099454 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c69ced50f30df5156ec7e67561d9f SystemUUID:ec2c69ce-d50f-30df-5156-ec7e67561d9f BootID:3a9a3864-1341-4820-b8e4-fbdea582fc0a Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e7:10:32:1f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e7:10:32:1f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:2e:2c:3f:c4:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 07:28:34.774161 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.774153 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 07:28:34.775327 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.774227 2580 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 07:28:34.777818 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.777779 2580 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 07:28:34.777979 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.777822 2580 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 07:28:34.778060 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.777985 2580 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 07:28:34.778060 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.777997 2580 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 07:28:34.778060 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.778049 2580 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 07:28:34.778164 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.778074 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 07:28:34.779404 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.779391 2580 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 07:28:34.783908 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.783894 2580 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 07:28:34.784033 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.784023 2580 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 07:28:34.784894 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.784883 2580 kubelet.go:493] "Attempting to sync node with API server" Oct 31 07:28:34.784955 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.784900 2580 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 07:28:34.784955 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.784916 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 07:28:34.784955 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.784929 2580 kubelet.go:405] "Adding apiserver pod source" Oct 31 07:28:34.784955 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.784941 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 07:28:34.785286 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.785267 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 07:28:34.787784 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.787768 2580 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 07:28:34.794603 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.794588 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 07:28:34.795518 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795506 2580 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 07:28:34.795558 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795536 2580 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 07:28:34.795690 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795680 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 07:28:34.795722 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795694 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 07:28:34.795722 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795701 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 07:28:34.795722 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795707 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 07:28:34.795722 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795713 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 07:28:34.795722 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.795719 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 07:28:34.796392 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.796383 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 07:28:34.796420 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.796392 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 07:28:34.796420 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.796407 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 07:28:34.796420 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.796413 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 07:28:34.796493 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.796425 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 07:28:34.796493 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.796434 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 07:28:34.797905 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.797895 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 07:28:34.797905 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.797904 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 07:28:34.801677 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.801664 2580 server.go:1263] "Started kubelet" Oct 31 07:28:34.801771 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.801744 2580 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 07:28:34.801846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.801761 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 07:28:34.801895 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.801871 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 07:28:34.802355 ip-10-0-140-6 systemd[1]: Started Kubernetes Kubelet. Oct 31 07:28:34.805269 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.805256 2580 server.go:317] "Adding debug handlers to kubelet server" Oct 31 07:28:34.806024 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.806009 2580 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 07:28:34.808475 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.808457 2580 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 07:28:34.810756 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.810740 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 07:28:34.812070 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812057 2580 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 07:28:34.812177 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.812150 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:34.812834 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812808 2580 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 07:28:34.812834 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812834 2580 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 07:28:34.812944 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812808 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 07:28:34.812944 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812907 2580 reconstruct.go:97] "Volume reconstruction finished" Oct 31 07:28:34.812944 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812913 2580 reconciler.go:29] "Reconciler: start to sync state" Oct 31 07:28:34.812944 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812917 2580 factory.go:55] Registering systemd factory Oct 31 07:28:34.812944 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.812933 2580 factory.go:223] Registration of the systemd container factory successfully Oct 31 07:28:34.813161 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.813146 2580 factory.go:153] Registering CRI-O factory Oct 31 07:28:34.813161 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.813161 2580 factory.go:223] Registration of the crio container factory successfully Oct 31 07:28:34.813262 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.813208 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 07:28:34.813262 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.813227 2580 factory.go:103] Registering Raw factory Oct 31 07:28:34.813262 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.813240 2580 manager.go:1199] Started watching for new ooms in manager Oct 31 07:28:34.815349 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.815327 2580 manager.go:319] Starting recovery of all containers Oct 31 07:28:34.828020 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.828002 2580 manager.go:324] Recovery completed Oct 31 07:28:34.830646 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.830619 2580 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 31 07:28:34.833680 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.833665 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.835444 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.835429 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.835513 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.835455 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.835513 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.835465 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:34.836077 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.836064 2580 cpu_manager.go:226] "Starting" policy="none" Oct 31 07:28:34.836077 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.836075 2580 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 07:28:34.836179 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.836091 2580 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 07:28:34.836504 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.836488 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-6.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 07:28:34.836590 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.836526 2580 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 07:28:34.836590 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.836564 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.837656 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.836496 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18e38ca481abd595 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-10-31 07:28:34.801636757 +0000 UTC m=+0.405482334,LastTimestamp:2026-10-31 07:28:34.801636757 +0000 UTC m=+0.405482334,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Oct 31 07:28:34.838272 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.838260 2580 policy_none.go:50] "Start" Oct 31 07:28:34.838334 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.838275 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 07:28:34.838334 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.838286 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 07:28:34.839224 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.839208 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.839288 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.839234 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.839288 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.839247 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:34.840751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.840738 2580 policy_none.go:44] "Start" Oct 31 07:28:34.844453 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.844433 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 07:28:34.852817 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.852760 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 07:28:34.852817 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.852724 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18e38ca483afb090 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-10-31 07:28:34.835443856 +0000 UTC m=+0.439289433,LastTimestamp:2026-10-31 07:28:34.835443856 +0000 UTC m=+0.439289433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Oct 31 07:28:34.852973 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.852817 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-6.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 07:28:34.853218 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.853192 2580 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.867172 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-942wl" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.868636 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18e38ca483afee8d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-10-31 07:28:34.835459725 +0000 UTC m=+0.439305300,LastTimestamp:2026-10-31 07:28:34.835459725 +0000 UTC m=+0.439305300,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.878353 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-942wl" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.878358 2580 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.880366 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881263 2580 manager.go:341] "Starting Device Plugin manager" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.881293 2580 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881304 2580 server.go:87] "Starting device plugin registration server" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881526 2580 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881537 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881575 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881592 2580 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881614 2580 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.881650 2580 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881732 2580 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881816 2580 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.881823 2580 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.882196 2580 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.882236 2580 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:34.890338 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:34.886312 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-6.ec2.internal.18e38ca483b0116d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-6.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-140-6.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-140-6.ec2.internal,},FirstTimestamp:2026-10-31 07:28:34.835468653 +0000 UTC m=+0.439314229,LastTimestamp:2026-10-31 07:28:34.835468653 +0000 UTC m=+0.439314229,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-6.ec2.internal,}" Oct 31 07:28:34.903738 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.903716 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:28:34.982081 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982001 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.982081 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982004 2580 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal"] Oct 31 07:28:34.982235 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982086 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.982923 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982899 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.983034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982931 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.983034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982942 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:34.983034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.982904 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.983034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.983008 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.983034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.983021 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:34.983193 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.983042 2580 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-6.ec2.internal" Oct 31 07:28:34.985156 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985144 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.985318 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:34.985360 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985334 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.985806 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985780 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.985869 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985805 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.985869 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985821 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.985869 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985827 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.985869 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985836 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:34.985869 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.985845 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:34.987936 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.987920 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:28:34.988038 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.987942 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:34.988555 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.988540 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:34.988628 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.988567 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:34.988628 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:34.988577 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:35.002541 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.002521 2580 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.002541 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.002541 2580 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-6.ec2.internal\": node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.013691 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.013670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8952dcaf440f7a0e696c251b7a42e952-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"8952dcaf440f7a0e696c251b7a42e952\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.013817 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.013696 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.013817 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.013714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.018696 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.018677 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-6.ec2.internal\" not found" node="ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.021260 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.021244 2580 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-6.ec2.internal\" not found" node="ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.052579 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.052562 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.114115 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.113940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.114196 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.114131 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.114196 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.114149 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8952dcaf440f7a0e696c251b7a42e952-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"8952dcaf440f7a0e696c251b7a42e952\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.114196 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.114180 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8952dcaf440f7a0e696c251b7a42e952-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"8952dcaf440f7a0e696c251b7a42e952\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.114196 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.114042 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.114333 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.114230 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.153070 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.153051 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.253813 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.253730 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.324511 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.324487 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.326656 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.326641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:28:35.354644 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.354618 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.457623 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.457592 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.558643 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.558576 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.659592 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.659557 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.740624 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.740300 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 07:28:35.740624 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.740522 2580 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 07:28:35.760049 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.760021 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.847652 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:35.847605 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8952dcaf440f7a0e696c251b7a42e952.slice/crio-9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6 WatchSource:0}: Error finding container 9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6: Status 404 returned error can't find the container with id 9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6 Oct 31 07:28:35.849482 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:35.849456 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3208be082c57abb809b875a8b99d7ba5.slice/crio-06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305 WatchSource:0}: Error finding container 06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305: Status 404 returned error can't find the container with id 06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305 Oct 31 07:28:35.852915 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.852885 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 07:28:35.860240 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:35.860224 2580 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:28:35.885348 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.885317 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:35 +0000 UTC" deadline="2028-04-28 17:18:25.225105638 +0000 UTC" Oct 31 07:28:35.885348 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.885347 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13089h49m49.339762028s" Oct 31 07:28:35.886009 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.885970 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"3208be082c57abb809b875a8b99d7ba5","Type":"ContainerStarted","Data":"06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305"} Oct 31 07:28:35.887052 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.887028 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" event={"ID":"8952dcaf440f7a0e696c251b7a42e952","Type":"ContainerStarted","Data":"9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6"} Oct 31 07:28:35.924147 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.924124 2580 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:28:35.925059 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.925043 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:28:35.925142 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.925070 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:28:35.925142 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.925080 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:28:35.925787 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:35.925772 2580 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:28:36.016340 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.016319 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:28:36.077080 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.077062 2580 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:28:36.077236 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.077220 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 07:28:36.079641 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.079625 2580 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:28:36.104848 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.104795 2580 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 07:28:36.423479 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.423392 2580 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:28:36.786330 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.786257 2580 apiserver.go:52] "Watching apiserver" Oct 31 07:28:36.812356 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.812323 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 07:28:36.822896 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.822869 2580 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 07:28:36.823278 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.823255 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-2qrrp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz","openshift-cluster-node-tuning-operator/tuned-wxjpf","openshift-image-registry/node-ca-m8msg","kube-system/global-pull-secret-syncer-l5c85","openshift-dns/node-resolver-sv7zk","openshift-multus/multus-qspqb","openshift-ovn-kubernetes/ovnkube-node-2b2kf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal","openshift-multus/multus-additional-cni-plugins-btjmv","openshift-multus/network-metrics-daemon-qnzx5","openshift-network-diagnostics/network-check-target-gdgxp","kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal","openshift-network-operator/iptables-alerter-wwhpb"] Oct 31 07:28:36.826128 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.826101 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:36.828747 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.828721 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.830912 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.830885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.833442 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.833408 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:36.840915 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.840887 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:36.843634 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.843616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:36.843747 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:36.843698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:36.846258 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.846239 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.847639 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.847618 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xcjxx\"" Oct 31 07:28:36.847736 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.847716 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.847802 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.847792 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9v74w\"" Oct 31 07:28:36.847857 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.847848 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.847947 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.847931 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.848008 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.847992 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 07:28:36.848063 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.848054 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.848840 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.848819 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.850926 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.850895 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qvcsq\"" Oct 31 07:28:36.851029 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.850929 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.851029 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.851002 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 07:28:36.851844 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.851508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.854816 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.854783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:36.854933 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:36.854882 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:36.856819 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.856796 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.857132 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.857097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:36.857304 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:36.857159 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:36.857304 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.857227 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 07:28:36.857304 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.857271 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 07:28:36.857515 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.857337 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 07:28:36.857515 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.857369 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ddkhs\"" Oct 31 07:28:36.859281 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.859264 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.859386 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.859306 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 07:28:36.859386 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.859346 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-krpl2\"" Oct 31 07:28:36.859386 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.859348 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.859745 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.859727 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:36.878288 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.878264 2580 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 07:28:36.878408 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.878381 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vt629" Oct 31 07:28:36.878408 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.878398 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vt629" Oct 31 07:28:36.914848 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.914796 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 07:28:36.924096 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-run\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.924211 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-lib-modules\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.924211 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924136 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:36.924211 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-kubelet\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.924211 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-netns\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-ovn\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924257 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-cnibin\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-system-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924305 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-cni-binary-copy\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924332 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-k8s-cni-cncf-io\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924352 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-kubelet\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924368 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-netns\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-bin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.924435 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-conf\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924514 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-var-lib-kubelet\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924567 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924594 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-device-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924618 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e12a075-b4ec-44e3-8811-7b19f1410513-host\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924643 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924689 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b163e33d-f1b0-4583-aeee-1e759841ef01-ovn-node-metrics-cert\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924727 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a122982a-c398-47f1-a368-fd89db8899a7-hosts-file\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924811 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.924876 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924881 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-script-lib\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-binary-copy\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.924980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-socket-dir-parent\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925007 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a122982a-c398-47f1-a368-fd89db8899a7-tmp-dir\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925031 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82xjk\" (UniqueName: \"kubernetes.io/projected/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kube-api-access-82xjk\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-konnectivity-ca\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925082 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-dbus\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925119 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-multus-certs\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-sys\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-host\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925241 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e12a075-b4ec-44e3-8811-7b19f1410513-serviceca\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925272 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-log-socket\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925315 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925309 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwmx\" (UniqueName: \"kubernetes.io/projected/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-kube-api-access-pwwmx\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925339 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb5h7\" (UniqueName: \"kubernetes.io/projected/b9dacaea-1664-495c-b164-a939a91bac87-kube-api-access-hb5h7\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925391 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-slash\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925425 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-systemd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925466 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-iptables-alerter-script\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925509 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-agent-certs\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-hostroot\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925648 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rqdj\" (UniqueName: \"kubernetes.io/projected/29d86a3e-2f5d-433b-835f-56df35aa18c2-kube-api-access-8rqdj\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysconfig\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-netd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-config\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-system-cni-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-conf-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.925846 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-etc-kubernetes\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-kubelet-config\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925940 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925972 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-daemon-config\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.925996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-systemd\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-etc-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6qvs\" (UniqueName: \"kubernetes.io/projected/a122982a-c398-47f1-a368-fd89db8899a7-kube-api-access-l6qvs\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926072 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-env-overrides\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqcq5\" (UniqueName: \"kubernetes.io/projected/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-kube-api-access-rqcq5\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926162 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-systemd-units\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-cnibin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926262 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-modprobe-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926324 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-var-lib-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.926466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926360 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-multus\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:36.927223 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-tmp\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.927223 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:36.927223 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.926965 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-host-slash\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:36.927223 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927074 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-node-log\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.927609 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-bin\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.927731 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927632 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rklq\" (UniqueName: \"kubernetes.io/projected/b163e33d-f1b0-4583-aeee-1e759841ef01-kube-api-access-8rklq\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:36.927731 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.927868 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p94ln\" (UniqueName: \"kubernetes.io/projected/078fca68-9d2f-4df5-924c-66d9c4198984-kube-api-access-p94ln\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.927868 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-kubernetes\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.927868 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927805 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkrd\" (UniqueName: \"kubernetes.io/projected/9e12a075-b4ec-44e3-8811-7b19f1410513-kube-api-access-8jkrd\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:36.928034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:36.928034 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.927914 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-tuned\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:36.953549 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.953529 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.953652 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.953583 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 07:28:36.953735 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.953711 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.953874 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.953807 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.954220 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954193 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mwmjb\"" Oct 31 07:28:36.954317 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954253 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:28:36.954378 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954333 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.954434 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954416 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 07:28:36.954521 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954507 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mrcbt\"" Oct 31 07:28:36.954699 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954682 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 07:28:36.954754 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954744 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 07:28:36.954804 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954783 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 07:28:36.954859 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954836 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 07:28:36.954958 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954944 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rhfwc\"" Oct 31 07:28:36.955013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.954987 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 07:28:36.955125 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.955106 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hr94j\"" Oct 31 07:28:36.955228 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.955152 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 07:28:36.993630 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:36.993603 2580 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:28:37.028457 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkrd\" (UniqueName: \"kubernetes.io/projected/9e12a075-b4ec-44e3-8811-7b19f1410513-kube-api-access-8jkrd\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.028586 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.028586 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028498 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-tuned\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.028586 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-run\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.028751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028584 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-lib-modules\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.028751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:37.028751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-kubelet\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.028751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028692 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-netns\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.028751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-ovn\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.028751 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-cnibin\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-system-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028801 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-cni-binary-copy\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028843 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-k8s-cni-cncf-io\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028869 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-kubelet\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028902 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-netns\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028894 2580 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028943 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-system-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.028975 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029005 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-ovn\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029018 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-lib-modules\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029050 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-cnibin\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-netns\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029083 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029088 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-bin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029095 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-kubelet\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029115 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029140 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-k8s-cni-cncf-io\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029141 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-conf\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029181 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-var-lib-kubelet\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029230 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-device-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e12a075-b4ec-44e3-8811-7b19f1410513-host\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029267 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-conf\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029274 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029305 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b163e33d-f1b0-4583-aeee-1e759841ef01-ovn-node-metrics-cert\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029318 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-bin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029333 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029347 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-device-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a122982a-c398-47f1-a368-fd89db8899a7-hosts-file\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029370 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-cni-binary-copy\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.029772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029399 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-var-lib-kubelet\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029422 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029429 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029445 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029458 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-netns\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029461 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-script-lib\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-run\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029485 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e12a075-b4ec-44e3-8811-7b19f1410513-host\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029491 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-binary-copy\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029521 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-socket-dir-parent\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a122982a-c398-47f1-a368-fd89db8899a7-tmp-dir\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.029557 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029593 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xjk\" (UniqueName: \"kubernetes.io/projected/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kube-api-access-82xjk\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.029629 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret podName:2c31ebf7-17d4-4aef-824b-afd78bd52d6e nodeName:}" failed. No retries permitted until 2026-10-31 07:28:37.529602314 +0000 UTC m=+3.133447899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret") pod "global-pull-secret-syncer-l5c85" (UID: "2c31ebf7-17d4-4aef-824b-afd78bd52d6e") : object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:37.030649 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-konnectivity-ca\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029678 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-dbus\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-multus-certs\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-sys\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029757 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-host\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029781 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-socket-dir-parent\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e12a075-b4ec-44e3-8811-7b19f1410513-serviceca\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-log-socket\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwmx\" (UniqueName: \"kubernetes.io/projected/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-kube-api-access-pwwmx\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5h7\" (UniqueName: \"kubernetes.io/projected/b9dacaea-1664-495c-b164-a939a91bac87-kube-api-access-hb5h7\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-slash\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029932 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-systemd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029960 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-iptables-alerter-script\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029965 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-sys\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030020 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-agent-certs\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-hostroot\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030066 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a122982a-c398-47f1-a368-fd89db8899a7-tmp-dir\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.031568 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030074 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqdj\" (UniqueName: \"kubernetes.io/projected/29d86a3e-2f5d-433b-835f-56df35aa18c2-kube-api-access-8rqdj\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029541 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a122982a-c398-47f1-a368-fd89db8899a7-hosts-file\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysconfig\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030160 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-dbus\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030166 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysconfig\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030161 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-host\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-multus-certs\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030370 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030074 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-binary-copy\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030638 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030653 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e12a075-b4ec-44e3-8811-7b19f1410513-serviceca\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.029464 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-kubelet\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030718 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-systemd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030715 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030759 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-konnectivity-ca\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030778 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-slash\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030804 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-log-socket\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-netd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.032488 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-config\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030882 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-system-cni-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-conf-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030915 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-hostroot\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030955 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-netd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.030975 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-conf-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031003 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-etc-kubernetes\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031036 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-kubelet-config\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031060 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-daemon-config\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-systemd\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031137 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-etc-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qvs\" (UniqueName: \"kubernetes.io/projected/a122982a-c398-47f1-a368-fd89db8899a7-kube-api-access-l6qvs\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-env-overrides\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031197 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-script-lib\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcq5\" (UniqueName: \"kubernetes.io/projected/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-kube-api-access-rqcq5\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031243 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-etc-kubernetes\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.033246 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-systemd-units\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031289 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-iptables-alerter-script\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-cnibin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031333 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-systemd\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031346 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-modprobe-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031379 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-etc-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031402 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031433 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-config\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031444 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-var-lib-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031454 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-cnibin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031478 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-multus\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031482 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-var-lib-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-tmp\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031521 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-multus\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031536 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-modprobe-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:37.034138 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031349 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-system-cni-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031562 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-systemd-units\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031580 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-host-slash\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-kubelet-config\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031618 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031652 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-node-log\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031671 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031612 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-node-log\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031713 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-bin\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031739 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rklq\" (UniqueName: \"kubernetes.io/projected/b163e33d-f1b0-4583-aeee-1e759841ef01-kube-api-access-8rklq\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031767 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p94ln\" (UniqueName: \"kubernetes.io/projected/078fca68-9d2f-4df5-924c-66d9c4198984-kube-api-access-p94ln\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-kubernetes\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.031922 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-kubernetes\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032025 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-env-overrides\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032084 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-host-slash\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032123 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-daemon-config\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.032158 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:37.035013 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032187 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.032208 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:37.532188706 +0000 UTC m=+3.136034288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032208 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-bin\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032462 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.032987 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-tuned\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.034024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-agent-certs\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.035139 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-tmp\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.035886 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.035180 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b163e33d-f1b0-4583-aeee-1e759841ef01-ovn-node-metrics-cert\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.053935 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.053855 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 07:28:37.053935 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.053882 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 07:28:37.053935 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.053895 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qnr8n for pod openshift-network-diagnostics/network-check-target-gdgxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:37.054153 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.053965 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n podName:a6b34ebf-4617-4662-ad56-5dbbb97a4e68 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:37.553950068 +0000 UTC m=+3.157795674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qnr8n" (UniqueName: "kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n") pod "network-check-target-gdgxp" (UID: "a6b34ebf-4617-4662-ad56-5dbbb97a4e68") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:37.058129 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.058103 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xjk\" (UniqueName: \"kubernetes.io/projected/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kube-api-access-82xjk\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.058323 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.058298 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5h7\" (UniqueName: \"kubernetes.io/projected/b9dacaea-1664-495c-b164-a939a91bac87-kube-api-access-hb5h7\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:37.058323 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.058314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkrd\" (UniqueName: \"kubernetes.io/projected/9e12a075-b4ec-44e3-8811-7b19f1410513-kube-api-access-8jkrd\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.058454 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.058391 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwmx\" (UniqueName: \"kubernetes.io/projected/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-kube-api-access-pwwmx\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.062746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.062719 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qvs\" (UniqueName: \"kubernetes.io/projected/a122982a-c398-47f1-a368-fd89db8899a7-kube-api-access-l6qvs\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.063255 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.063235 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqdj\" (UniqueName: \"kubernetes.io/projected/29d86a3e-2f5d-433b-835f-56df35aa18c2-kube-api-access-8rqdj\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.064002 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.063976 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rklq\" (UniqueName: \"kubernetes.io/projected/b163e33d-f1b0-4583-aeee-1e759841ef01-kube-api-access-8rklq\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.064108 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.064044 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcq5\" (UniqueName: \"kubernetes.io/projected/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-kube-api-access-rqcq5\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.064845 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.064809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94ln\" (UniqueName: \"kubernetes.io/projected/078fca68-9d2f-4df5-924c-66d9c4198984-kube-api-access-p94ln\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.143456 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.143425 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:28:37.158944 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.158919 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:28:37.168309 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.168283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qspqb" Oct 31 07:28:37.171523 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.171496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:37.174163 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.174123 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda122982a_c398_47f1_a368_fd89db8899a7.slice/crio-86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e WatchSource:0}: Error finding container 86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e: Status 404 returned error can't find the container with id 86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e Oct 31 07:28:37.176619 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.175550 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8cbe30_d117_4aa4_92ed_e7e81544a1f4.slice/crio-8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857 WatchSource:0}: Error finding container 8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857: Status 404 returned error can't find the container with id 8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857 Oct 31 07:28:37.180539 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.180392 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:28:37.183625 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.183600 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d86a3e_2f5d_433b_835f_56df35aa18c2.slice/crio-874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71 WatchSource:0}: Error finding container 874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71: Status 404 returned error can't find the container with id 874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71 Oct 31 07:28:37.187326 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.187303 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb684428_62fd_4dc0_aa16_3ba4dfc7c89b.slice/crio-3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85 WatchSource:0}: Error finding container 3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85: Status 404 returned error can't find the container with id 3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85 Oct 31 07:28:37.191197 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.191178 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:28:37.193283 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.193255 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e12a075_b4ec_44e3_8811_7b19f1410513.slice/crio-7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78 WatchSource:0}: Error finding container 7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78: Status 404 returned error can't find the container with id 7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78 Oct 31 07:28:37.204345 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.204317 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52ee83a_3e9c_49e9_b599_ec9a43d40bde.slice/crio-0350a9e66af9f1414ebc1ad97da2afc92553c432a1f214b5e26c9cbd266cfef2 WatchSource:0}: Error finding container 0350a9e66af9f1414ebc1ad97da2afc92553c432a1f214b5e26c9cbd266cfef2: Status 404 returned error can't find the container with id 0350a9e66af9f1414ebc1ad97da2afc92553c432a1f214b5e26c9cbd266cfef2 Oct 31 07:28:37.219982 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.219954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:37.222158 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.222139 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:28:37.232220 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.232184 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb163e33d_f1b0_4583_aeee_1e759841ef01.slice/crio-40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615 WatchSource:0}: Error finding container 40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615: Status 404 returned error can't find the container with id 40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615 Oct 31 07:28:37.236305 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.236280 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:28:37.248351 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:28:37.248326 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078fca68_9d2f_4df5_924c_66d9c4198984.slice/crio-40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c WatchSource:0}: Error finding container 40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c: Status 404 returned error can't find the container with id 40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c Oct 31 07:28:37.534409 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.534376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:37.534533 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.534465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:37.534644 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.534623 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:37.534772 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.534693 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:38.534677796 +0000 UTC m=+4.138523365 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:37.534772 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.534695 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:37.534772 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.534748 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret podName:2c31ebf7-17d4-4aef-824b-afd78bd52d6e nodeName:}" failed. No retries permitted until 2026-10-31 07:28:38.534732207 +0000 UTC m=+4.138577773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret") pod "global-pull-secret-syncer-l5c85" (UID: "2c31ebf7-17d4-4aef-824b-afd78bd52d6e") : object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:37.635776 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.635699 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:37.635950 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.635904 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 07:28:37.635950 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.635929 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 07:28:37.635950 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.635942 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qnr8n for pod openshift-network-diagnostics/network-check-target-gdgxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:37.636066 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:37.635999 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n podName:a6b34ebf-4617-4662-ad56-5dbbb97a4e68 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:38.635980579 +0000 UTC m=+4.239826152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnr8n" (UniqueName: "kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n") pod "network-check-target-gdgxp" (UID: "a6b34ebf-4617-4662-ad56-5dbbb97a4e68") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:37.882055 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.881934 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:37 +0000 UTC" deadline="2028-07-12 02:33:20.94327158 +0000 UTC" Oct 31 07:28:37.882055 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.882000 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14875h4m43.061275632s" Oct 31 07:28:37.912905 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.912716 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerStarted","Data":"40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c"} Oct 31 07:28:37.919734 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.919698 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615"} Oct 31 07:28:37.935352 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.935254 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" event={"ID":"a52ee83a-3e9c-49e9-b599-ec9a43d40bde","Type":"ContainerStarted","Data":"0350a9e66af9f1414ebc1ad97da2afc92553c432a1f214b5e26c9cbd266cfef2"} Oct 31 07:28:37.942779 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.942741 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m8msg" event={"ID":"9e12a075-b4ec-44e3-8811-7b19f1410513","Type":"ContainerStarted","Data":"7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78"} Oct 31 07:28:37.959004 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.958972 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2qrrp" event={"ID":"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b","Type":"ContainerStarted","Data":"3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85"} Oct 31 07:28:37.970061 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.970028 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" event={"ID":"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4","Type":"ContainerStarted","Data":"8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857"} Oct 31 07:28:37.974788 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.973662 2580 generic.go:356] "Generic (PLEG): container finished" podID="3208be082c57abb809b875a8b99d7ba5" containerID="5bb838a3df977cd37a557a0563db09cbc7dfa34f5359aeefff920eb05f34e779" exitCode=0 Oct 31 07:28:37.974788 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.974540 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"3208be082c57abb809b875a8b99d7ba5","Type":"ContainerDied","Data":"5bb838a3df977cd37a557a0563db09cbc7dfa34f5359aeefff920eb05f34e779"} Oct 31 07:28:37.983573 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:37.983492 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wwhpb" event={"ID":"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5","Type":"ContainerStarted","Data":"306bb57eb3ae4766b2515fc3efe62bdccc18cffe818844e2c53028c98d654a48"} Oct 31 07:28:38.001599 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.001514 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qspqb" event={"ID":"29d86a3e-2f5d-433b-835f-56df35aa18c2","Type":"ContainerStarted","Data":"874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71"} Oct 31 07:28:38.019925 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.019889 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sv7zk" event={"ID":"a122982a-c398-47f1-a368-fd89db8899a7","Type":"ContainerStarted","Data":"86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e"} Oct 31 07:28:38.045055 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.045018 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" event={"ID":"8952dcaf440f7a0e696c251b7a42e952","Type":"ContainerStarted","Data":"79e52bf58fe37bb1c959dee4a387635ac916c00515ca8cc1f64dd18a2c779481"} Oct 31 07:28:38.544314 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.543466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:38.544314 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.543544 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:38.544314 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.543682 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:38.544314 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.543745 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret podName:2c31ebf7-17d4-4aef-824b-afd78bd52d6e nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.543726597 +0000 UTC m=+6.147572178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret") pod "global-pull-secret-syncer-l5c85" (UID: "2c31ebf7-17d4-4aef-824b-afd78bd52d6e") : object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:38.544314 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.544177 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:38.544314 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.544231 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.544213526 +0000 UTC m=+6.148059113 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:38.644182 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.644143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:38.644387 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.644369 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 07:28:38.644387 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.644395 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 07:28:38.644536 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.644409 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qnr8n for pod openshift-network-diagnostics/network-check-target-gdgxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:38.644536 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.644473 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n podName:a6b34ebf-4617-4662-ad56-5dbbb97a4e68 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:40.644448469 +0000 UTC m=+6.248294038 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnr8n" (UniqueName: "kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n") pod "network-check-target-gdgxp" (UID: "a6b34ebf-4617-4662-ad56-5dbbb97a4e68") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:38.885301 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.884351 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:38.885301 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.884482 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:38.885301 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.884906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:38.885301 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.885015 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:38.885301 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.885076 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:38.885301 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:38.885146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:38.886233 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.886162 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:37 +0000 UTC" deadline="2028-07-14 12:42:57.220282334 +0000 UTC" Oct 31 07:28:38.886233 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:38.886196 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14933h14m18.334091482s" Oct 31 07:28:39.062150 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:39.062113 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"3208be082c57abb809b875a8b99d7ba5","Type":"ContainerStarted","Data":"defc31f8355364286d8f2480338e060715a8252d5cf93daa01d87eecd3989a6d"} Oct 31 07:28:39.110956 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:39.110889 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" podStartSLOduration=3.110869589 podStartE2EDuration="3.110869589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:39.110486142 +0000 UTC m=+4.714331738" watchObservedRunningTime="2026-10-31 07:28:39.110869589 +0000 UTC m=+4.714715182" Oct 31 07:28:39.111260 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:39.111209 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" podStartSLOduration=3.111201484 podStartE2EDuration="3.111201484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:38.093075475 +0000 UTC m=+3.696921064" watchObservedRunningTime="2026-10-31 07:28:39.111201484 +0000 UTC m=+4.715047073" Oct 31 07:28:40.560612 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:40.560524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:40.560612 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:40.560611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:40.561260 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.560766 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:40.561260 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.560821 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret podName:2c31ebf7-17d4-4aef-824b-afd78bd52d6e nodeName:}" failed. No retries permitted until 2026-10-31 07:28:44.560807302 +0000 UTC m=+10.164652882 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret") pod "global-pull-secret-syncer-l5c85" (UID: "2c31ebf7-17d4-4aef-824b-afd78bd52d6e") : object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:40.561260 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.561240 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:40.561433 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.561291 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:44.561273647 +0000 UTC m=+10.165119219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:40.661692 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:40.661652 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:40.661894 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.661865 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 07:28:40.661894 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.661888 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 07:28:40.662023 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.661902 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qnr8n for pod openshift-network-diagnostics/network-check-target-gdgxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:40.662023 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.661970 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n podName:a6b34ebf-4617-4662-ad56-5dbbb97a4e68 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:44.661951767 +0000 UTC m=+10.265797340 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnr8n" (UniqueName: "kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n") pod "network-check-target-gdgxp" (UID: "a6b34ebf-4617-4662-ad56-5dbbb97a4e68") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:40.888291 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:40.888186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:40.888458 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.888320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:40.888778 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:40.888738 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:40.888893 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.888848 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:40.889025 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:40.888960 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:40.889096 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:40.889058 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:41.089211 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:41.089171 2580 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="711ae334f61129f3d8eab456920d7620c9c1ed35d3b23956624581c22fca4340" exitCode=0 Oct 31 07:28:41.089380 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:41.089236 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerDied","Data":"711ae334f61129f3d8eab456920d7620c9c1ed35d3b23956624581c22fca4340"} Oct 31 07:28:42.097759 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:42.097032 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2qrrp" event={"ID":"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b","Type":"ContainerStarted","Data":"19f30b8a0af507ca487d16e40af70389434dee8a81a3e3611e039f5d513c2dce"} Oct 31 07:28:42.892340 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:42.892142 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:42.892340 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:42.892137 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:42.892597 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:42.892366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:42.892913 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:42.892752 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:42.893510 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:42.893049 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:42.893510 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:42.893480 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:43.109581 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:43.109544 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m8msg" event={"ID":"9e12a075-b4ec-44e3-8811-7b19f1410513","Type":"ContainerStarted","Data":"b09fa980c8eaf043c576ea385f01594eff94e7e0efe6874c3a9f65aa1aa7b83c"} Oct 31 07:28:43.161153 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:43.159950 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2qrrp" podStartSLOduration=4.202451141 podStartE2EDuration="8.159930673s" totalImagesPullingTime="3.957479532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:42.147327724 +0000 UTC m=+7.751173315" watchObservedRunningTime="2026-10-31 07:28:43.159930673 +0000 UTC m=+8.763776264" Oct 31 07:28:44.119688 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.119632 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" event={"ID":"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4","Type":"ContainerStarted","Data":"417539d14aeb08e27313b9d613c4c6f5077bda1db7021f58423cdcf6bc45bf62"} Oct 31 07:28:44.598752 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.598725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:44.598943 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.598780 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:44.598943 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.598900 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:44.599043 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.598968 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:52.598947971 +0000 UTC m=+18.202793564 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:44.599043 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.598904 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:44.599043 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.599042 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret podName:2c31ebf7-17d4-4aef-824b-afd78bd52d6e nodeName:}" failed. No retries permitted until 2026-10-31 07:28:52.599029285 +0000 UTC m=+18.202874857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret") pod "global-pull-secret-syncer-l5c85" (UID: "2c31ebf7-17d4-4aef-824b-afd78bd52d6e") : object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:44.700563 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.699107 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:44.700563 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.699310 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 07:28:44.700563 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.699332 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 07:28:44.700563 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.699348 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qnr8n for pod openshift-network-diagnostics/network-check-target-gdgxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:44.700563 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.699406 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n podName:a6b34ebf-4617-4662-ad56-5dbbb97a4e68 nodeName:}" failed. No retries permitted until 2026-10-31 07:28:52.699386444 +0000 UTC m=+18.303232017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnr8n" (UniqueName: "kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n") pod "network-check-target-gdgxp" (UID: "a6b34ebf-4617-4662-ad56-5dbbb97a4e68") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:44.742534 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.742507 2580 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 07:28:44.788628 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.788581 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:44.886657 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.886593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:44.886786 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.886685 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:44.887058 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.887037 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:44.887177 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.887156 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:44.887221 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.887209 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:44.887305 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:44.887291 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:44.887910 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.887782 2580 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:28:44.742527876Z","UUID":"23c0f8aa-72f2-42c9-a1e7-3eeba25c8655","Handler":null,"Name":"","Endpoint":""} Oct 31 07:28:44.891144 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.891118 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 07:28:44.891275 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.891167 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 07:28:44.902243 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.902178 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:44.903657 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:44.903629 2580 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:45.001814 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:45.001721 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m8msg" podStartSLOduration=3.836134295 podStartE2EDuration="9.001701405s" totalImagesPullingTime="5.16556711s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:43.161083366 +0000 UTC m=+8.764928958" watchObservedRunningTime="2026-10-31 07:28:45.001701405 +0000 UTC m=+10.605546995" Oct 31 07:28:45.125258 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:45.125137 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" event={"ID":"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4","Type":"ContainerStarted","Data":"18c71eeb7fc3ec7407c92422bcde841d2b7441e08fa78508a762d8fed3650f5b"} Oct 31 07:28:45.126964 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:45.126933 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:28:46.129348 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.129166 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qspqb" event={"ID":"29d86a3e-2f5d-433b-835f-56df35aa18c2","Type":"ContainerStarted","Data":"2e66d18ebc3b13f4f165df188f1557a420940eba91b0173096dd835213d22b80"} Oct 31 07:28:46.131325 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.131287 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-sv7zk" event={"ID":"a122982a-c398-47f1-a368-fd89db8899a7","Type":"ContainerStarted","Data":"35caec16246698f4570853da8e472e7aa6e462a40ce45a9325e1ff77d85d8b5e"} Oct 31 07:28:46.133475 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.133418 2580 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="67a826f450028581be5d153374894f56b01f7e53dac3c73b76ce8fbbd2dad779" exitCode=0 Oct 31 07:28:46.133566 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.133518 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerDied","Data":"67a826f450028581be5d153374894f56b01f7e53dac3c73b76ce8fbbd2dad779"} Oct 31 07:28:46.137447 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.137414 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" event={"ID":"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4","Type":"ContainerStarted","Data":"9f582f5d0c473b6da7599b1c302eb9d15cbca6b490b9a2b6370e58e2119d3ff5"} Oct 31 07:28:46.181062 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.181014 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-qspqb" podStartSLOduration=3.055434495 podStartE2EDuration="11.181001813s" totalImagesPullingTime="8.125567318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:46.180341389 +0000 UTC m=+11.784186978" watchObservedRunningTime="2026-10-31 07:28:46.181001813 +0000 UTC m=+11.784847402" Oct 31 07:28:46.281494 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.281434 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-sv7zk" podStartSLOduration=3.527454027 podStartE2EDuration="11.281420582s" totalImagesPullingTime="7.753966555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:46.280963529 +0000 UTC m=+11.884809121" watchObservedRunningTime="2026-10-31 07:28:46.281420582 +0000 UTC m=+11.885266171" Oct 31 07:28:46.332006 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.331945 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" podStartSLOduration=3.02973456 podStartE2EDuration="11.33192703s" totalImagesPullingTime="8.30219247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:46.33164515 +0000 UTC m=+11.935490742" watchObservedRunningTime="2026-10-31 07:28:46.33192703 +0000 UTC m=+11.935772617" Oct 31 07:28:46.882116 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.881900 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:46.882261 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:46.882231 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:46.882613 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.882593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:46.882727 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:46.882704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:46.882945 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:46.882924 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:46.883052 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:46.883011 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:47.142312 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:47.142277 2580 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="5bf27ca0c80a224c76b95940b677516114f5e52f37cd45be8ff303bed7ec0e23" exitCode=0 Oct 31 07:28:47.142742 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:47.142361 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerDied","Data":"5bf27ca0c80a224c76b95940b677516114f5e52f37cd45be8ff303bed7ec0e23"} Oct 31 07:28:47.149966 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:47.149934 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wwhpb" event={"ID":"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5","Type":"ContainerStarted","Data":"be15588eba334e64c2f27325cbfb035eb68cf25819ece9d7e73d49851372c182"} Oct 31 07:28:47.246161 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:47.245860 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wwhpb" podStartSLOduration=2.923575108 podStartE2EDuration="11.245841156s" totalImagesPullingTime="8.322266048s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:47.244884181 +0000 UTC m=+12.848729769" watchObservedRunningTime="2026-10-31 07:28:47.245841156 +0000 UTC m=+12.849686743" Oct 31 07:28:48.154750 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.154716 2580 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="da07b725a9e32d6f1cd6b68c0e94861efb60fc5d6da554c1cf6fd080e18c8d20" exitCode=0 Oct 31 07:28:48.155280 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.154825 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerDied","Data":"da07b725a9e32d6f1cd6b68c0e94861efb60fc5d6da554c1cf6fd080e18c8d20"} Oct 31 07:28:48.158280 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.158257 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" event={"ID":"a52ee83a-3e9c-49e9-b599-ec9a43d40bde","Type":"ContainerStarted","Data":"a336f685ce983765374cd8f40f7393b66e5d5a1d75b9071684993d08c9f78f34"} Oct 31 07:28:48.267181 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.267138 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" podStartSLOduration=1.4402678340000001 podStartE2EDuration="12.267124679s" totalImagesPullingTime="10.826856845s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:48.266697422 +0000 UTC m=+13.870543009" watchObservedRunningTime="2026-10-31 07:28:48.267124679 +0000 UTC m=+13.870970265" Oct 31 07:28:48.886247 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.886220 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:48.886420 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.886259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:48.886420 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:48.886353 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:48.886825 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:48.886782 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:48.886927 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:48.886867 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:48.886975 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:48.886959 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:50.165508 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:50.165472 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerStarted","Data":"3cb3c765f6668039d7e6fc85b990d843569a357319f52ed71eb03b3914e5e5e1"} Oct 31 07:28:50.885897 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:50.885865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:50.885897 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:50.885906 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:50.886094 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:50.885937 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:50.886094 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:50.886036 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:50.886212 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:50.886107 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:50.886212 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:50.886193 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:51.171047 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:51.170976 2580 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="3cb3c765f6668039d7e6fc85b990d843569a357319f52ed71eb03b3914e5e5e1" exitCode=0 Oct 31 07:28:51.171047 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:51.171030 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerDied","Data":"3cb3c765f6668039d7e6fc85b990d843569a357319f52ed71eb03b3914e5e5e1"} Oct 31 07:28:52.176002 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.175812 2580 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="4fe3aa1422601672dcaa52ee2d0365c94885958a5482699f9a2ca4c1df83bc9d" exitCode=0 Oct 31 07:28:52.176002 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.175890 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerDied","Data":"4fe3aa1422601672dcaa52ee2d0365c94885958a5482699f9a2ca4c1df83bc9d"} Oct 31 07:28:52.661963 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.661933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:52.662093 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.661986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:52.662093 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.662085 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:52.662205 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.662099 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:52.662205 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.662130 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret podName:2c31ebf7-17d4-4aef-824b-afd78bd52d6e nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.662119087 +0000 UTC m=+34.265964651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret") pod "global-pull-secret-syncer-l5c85" (UID: "2c31ebf7-17d4-4aef-824b-afd78bd52d6e") : object "kube-system"/"original-pull-secret" not registered Oct 31 07:28:52.662205 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.662159 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.662141585 +0000 UTC m=+34.265987168 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 07:28:52.762809 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.762720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:52.762918 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.762858 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 07:28:52.762918 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.762892 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 07:28:52.762918 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.762901 2580 projected.go:196] Error preparing data for projected volume kube-api-access-qnr8n for pod openshift-network-diagnostics/network-check-target-gdgxp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:52.763034 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.762942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n podName:a6b34ebf-4617-4662-ad56-5dbbb97a4e68 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:08.762930711 +0000 UTC m=+34.366776275 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qnr8n" (UniqueName: "kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n") pod "network-check-target-gdgxp" (UID: "a6b34ebf-4617-4662-ad56-5dbbb97a4e68") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 07:28:52.882923 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.882891 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:52.882923 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.882915 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:52.883076 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:52.882969 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:52.883076 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.883060 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:52.883193 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.883171 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:52.883259 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:52.883245 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:53.181687 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.181608 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-btjmv" event={"ID":"078fca68-9d2f-4df5-924c-66d9c4198984","Type":"ContainerStarted","Data":"eadf88fa9ce4c8d9fcc52a7ed8a527a7f3be4ba58fc0c871dfb2fbf466545e64"} Oct 31 07:28:53.184451 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.184428 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"4b89c9b2501cae2943f56f7e77316efc1df79633bfdc314ab6339300ec0d9b3c"} Oct 31 07:28:53.184555 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.184483 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"8bc3d56552a9738fd410618f35d5e0ce2f7ca5440a1f8cf56d60bb5dc3169b14"} Oct 31 07:28:53.184555 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.184512 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"0e679041e76fadf9754302694821bc5f01a784d6021045810bdffbcb4c26a3a4"} Oct 31 07:28:53.184555 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.184539 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"09c0dc0c1d08f0d13397b008742ba16d3ea36be3ce4a80dc5b518a5cb24ed944"} Oct 31 07:28:53.184642 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.184565 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"5785735258b2b8f8998909071f4fb1b4ecd81555be06d0d6ee20f51bc29ae0e9"} Oct 31 07:28:53.184642 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.184592 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"eb06bb769d735b79d3a24cdc872e1b96f0ddb0751ae89538af991059617e2c59"} Oct 31 07:28:53.232752 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:53.232708 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-btjmv" podStartSLOduration=6.750757365 podStartE2EDuration="17.232695974s" totalImagesPullingTime="10.481938609s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:53.232416945 +0000 UTC m=+18.836262606" watchObservedRunningTime="2026-10-31 07:28:53.232695974 +0000 UTC m=+18.836541560" Oct 31 07:28:54.883290 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:54.883216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:54.883641 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:54.883306 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:54.883641 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:54.883398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:54.883641 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:54.883495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:54.883641 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:54.883546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:54.883641 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:54.883596 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:55.192095 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:55.192016 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"12a2afd105c1c3e5590ce6d42cd73e6b2dc62cbfa49bde23c58689decc263ded"} Oct 31 07:28:56.882665 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:56.882508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:56.882665 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:56.882508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:56.882665 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:56.882659 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:56.883661 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:56.882764 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:56.883661 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:56.882781 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:56.883661 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:56.882892 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:57.201573 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.201485 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" event={"ID":"b163e33d-f1b0-4583-aeee-1e759841ef01","Type":"ContainerStarted","Data":"e03c41ef039a5bf4a2dcadbb53b52675eb6f3c61f678adf33eebd117c4df52b9"} Oct 31 07:28:57.201732 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.201714 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:57.201809 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.201779 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:57.216765 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.216742 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:57.216911 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.216844 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:57.303553 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.303384 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" podStartSLOduration=6.278505011 podStartE2EDuration="21.303372853s" totalImagesPullingTime="15.024867842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:28:57.288103374 +0000 UTC m=+22.891948961" watchObservedRunningTime="2026-10-31 07:28:57.303372853 +0000 UTC m=+22.907218457" Oct 31 07:28:57.496236 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:57.496138 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:28:58.802345 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:58.802311 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5c85"] Oct 31 07:28:58.802865 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:58.802441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:28:58.802865 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:58.802534 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:28:58.804810 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:58.804777 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gdgxp"] Oct 31 07:28:58.804918 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:58.804878 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:58.804960 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:58.804938 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:58.816681 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:58.816661 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qnzx5"] Oct 31 07:28:58.816770 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:58.816743 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:58.816847 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:58.816829 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:28:59.882471 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:59.882441 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:28:59.882881 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:59.882537 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:28:59.882881 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:28:59.882590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:28:59.882881 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:28:59.882681 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:29:00.885707 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:00.885682 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:00.886092 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:00.885776 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:29:01.882113 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:01.882042 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:01.882250 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:01.882041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:01.882250 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:01.882146 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:29:01.882250 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:01.882206 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:29:02.881772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:02.881741 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:02.882151 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:02.881893 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:29:03.881833 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:03.881779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:03.882339 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:03.881925 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gdgxp" podUID="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" Oct 31 07:29:03.882339 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:03.881972 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:03.882339 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:03.882097 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qnzx5" podUID="b9dacaea-1664-495c-b164-a939a91bac87" Oct 31 07:29:04.883572 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:04.883542 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:04.883949 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:04.883626 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-l5c85" podUID="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" Oct 31 07:29:05.205079 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.205055 2580 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeReady" Oct 31 07:29:05.205225 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.205177 2580 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 07:29:05.271962 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.271917 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xzc9s"] Oct 31 07:29:05.275512 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.275486 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:05.275836 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.275810 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk"] Oct 31 07:29:05.278865 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.278842 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57d7689f6c-h8wmf"] Oct 31 07:29:05.278990 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.278970 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.281696 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.281676 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6sww9"] Oct 31 07:29:05.281813 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.281778 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.284464 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.284447 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v5stm"] Oct 31 07:29:05.284591 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.284565 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:05.287626 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.287602 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-p9pnl"] Oct 31 07:29:05.287732 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.287716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:05.291160 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.291142 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-546c4cd566-xfkx2"] Oct 31 07:29:05.291402 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.291377 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:05.294924 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.294899 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vvm6c"] Oct 31 07:29:05.295994 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.295277 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.298603 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.298582 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc"] Oct 31 07:29:05.298702 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.298686 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.302830 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.302808 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.302926 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.302851 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp"] Oct 31 07:29:05.303022 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.302997 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.304091 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.304072 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.309818 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.307883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:05.309818 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.307766 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b4c78c6d5-pmxm4"] Oct 31 07:29:05.316383 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.314906 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj"] Oct 31 07:29:05.316383 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.315173 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.319924 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.319902 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc"] Oct 31 07:29:05.320026 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.320009 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.323225 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.323208 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5"] Oct 31 07:29:05.323510 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.323494 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.325951 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.325935 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hzng6"] Oct 31 07:29:05.326064 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.326047 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:05.329601 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.329583 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 07:29:05.329853 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.329836 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 07:29:05.330878 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.330048 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-g29bz"] Oct 31 07:29:05.330878 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.330204 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.342279 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.341335 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m99x4"] Oct 31 07:29:05.342505 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.342424 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.344672 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344650 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xzc9s"] Oct 31 07:29:05.344767 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344689 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk"] Oct 31 07:29:05.344767 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344711 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57d7689f6c-h8wmf"] Oct 31 07:29:05.344767 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344733 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-p9pnl"] Oct 31 07:29:05.344767 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344753 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp"] Oct 31 07:29:05.344995 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344775 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v5stm"] Oct 31 07:29:05.344995 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344821 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b4c78c6d5-pmxm4"] Oct 31 07:29:05.344995 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344842 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546c4cd566-xfkx2"] Oct 31 07:29:05.344995 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344889 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g29bz"] Oct 31 07:29:05.344995 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344911 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj"] Oct 31 07:29:05.344995 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.344943 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vvm6c"] Oct 31 07:29:05.345277 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.345026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.345327 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.345288 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6sww9"] Oct 31 07:29:05.353183 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.353162 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m99x4"] Oct 31 07:29:05.359052 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.359032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/805181c8-c83e-433d-9980-62f629c63f0d-serving-cert\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.359176 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.359159 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5j8\" (UniqueName: \"kubernetes.io/projected/805181c8-c83e-433d-9980-62f629c63f0d-kube-api-access-7q5j8\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.359214 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.359195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfzg\" (UniqueName: \"kubernetes.io/projected/35dff775-deae-46eb-a3bc-b5618173818f-kube-api-access-vbfzg\") pod \"kas-connection-checker-685895c7c9-xzc9s\" (UID: \"35dff775-deae-46eb-a3bc-b5618173818f\") " pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:05.359268 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.359248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/805181c8-c83e-433d-9980-62f629c63f0d-config\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.369803 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.369010 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5"] Oct 31 07:29:05.372081 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.371873 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc"] Oct 31 07:29:05.372254 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.372238 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc"] Oct 31 07:29:05.372591 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.372573 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hzng6"] Oct 31 07:29:05.402527 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.402449 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.402527 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.402512 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.402527 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.402512 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 07:29:05.402783 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.402768 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.402857 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.402845 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.402913 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.402903 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.403020 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403010 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.403084 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403036 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.403084 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403080 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 07:29:05.403222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403086 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.403222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403109 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.403222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403152 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 07:29:05.403222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403164 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 07:29:05.403222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403198 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.403222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403224 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.403519 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403251 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.403519 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403255 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 07:29:05.403519 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403274 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 07:29:05.403759 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403738 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 07:29:05.403848 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.403828 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.404077 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.404063 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.404128 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.404078 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 07:29:05.404418 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.404403 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.404516 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.404501 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.404600 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.404584 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.411010 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.410994 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.411269 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.411252 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:05.411316 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.411271 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:05.418652 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.418633 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 07:29:05.419048 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419031 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419059 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419069 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419066 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419078 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419093 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419097 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419109 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419034 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419112 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419129 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 07:29:05.419135 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419137 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 07:29:05.419490 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419147 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 07:29:05.419490 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419358 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-w856f\"" Oct 31 07:29:05.419490 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.419398 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jhgwz\"" Oct 31 07:29:05.426399 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.426385 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cp4x6\"" Oct 31 07:29:05.426464 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.426390 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvd7d\"" Oct 31 07:29:05.434403 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.434383 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 07:29:05.434911 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.434894 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 07:29:05.434911 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.434904 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 07:29:05.435281 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.435262 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:05.460384 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460325 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7qq\" (UniqueName: \"kubernetes.io/projected/92342503-edc9-45e9-b9d7-fff71ce5092d-kube-api-access-9s7qq\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.460384 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-service-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.460384 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8039f97e-ddca-4960-80a9-b53c24950ca9-tmp-dir\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.460532 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8039f97e-ddca-4960-80a9-b53c24950ca9-config-volume\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.460532 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.460590 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460555 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-default-certificate\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.460590 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460576 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfzg\" (UniqueName: \"kubernetes.io/projected/35dff775-deae-46eb-a3bc-b5618173818f-kube-api-access-vbfzg\") pod \"kas-connection-checker-685895c7c9-xzc9s\" (UID: \"35dff775-deae-46eb-a3bc-b5618173818f\") " pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:05.460661 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/805181c8-c83e-433d-9980-62f629c63f0d-config\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.460661 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brsk7\" (UniqueName: \"kubernetes.io/projected/a36d3876-b5ff-48cd-913c-48108ed86f77-kube-api-access-brsk7\") pod \"network-check-source-6f78d59f99-ll9q5\" (UID: \"a36d3876-b5ff-48cd-913c-48108ed86f77\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:05.460661 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460628 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.460661 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.460811 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92342503-edc9-45e9-b9d7-fff71ce5092d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.460855 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460817 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20fefbf1-a4ae-4208-a927-2dc47132a88c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.460897 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460865 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.460897 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.460996 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.460978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wjv9\" (UniqueName: \"kubernetes.io/projected/8f52e908-ef31-4672-ac4d-1ee9778d1b52-kube-api-access-9wjv9\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.461067 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz678\" (UniqueName: \"kubernetes.io/projected/b67627fb-3093-4fff-9ab1-c0a4432f72e2-kube-api-access-lz678\") pod \"kas-connection-checker-685895c7c9-v5stm\" (UID: \"b67627fb-3093-4fff-9ab1-c0a4432f72e2\") " pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:05.461067 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461044 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71bf711-37d2-421a-8bb5-a838f60d36dc-cert\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.461159 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.461159 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461140 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndmz\" (UniqueName: \"kubernetes.io/projected/5abb2415-7417-4478-9d12-a4d9f2e0549c-kube-api-access-jndmz\") pod \"volume-data-source-validator-58c9f5f46d-cqqrp\" (UID: \"5abb2415-7417-4478-9d12-a4d9f2e0549c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:05.461259 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461182 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f52e908-ef31-4672-ac4d-1ee9778d1b52-serving-cert\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.461259 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20fefbf1-a4ae-4208-a927-2dc47132a88c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.461403 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.461403 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461268 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/805181c8-c83e-433d-9980-62f629c63f0d-config\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.461403 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461354 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.461403 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs699\" (UniqueName: \"kubernetes.io/projected/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e-kube-api-access-hs699\") pod \"kas-connection-checker-685895c7c9-6sww9\" (UID: \"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e\") " pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:05.461403 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461399 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r2jl\" (UniqueName: \"kubernetes.io/projected/e71bf711-37d2-421a-8bb5-a838f60d36dc-kube-api-access-4r2jl\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.461598 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461418 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.461598 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461441 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.461598 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461476 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.461598 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.461598 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461588 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fec1a19-aa94-423d-a573-71c8b0739f55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:05.461785 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461662 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5j8\" (UniqueName: \"kubernetes.io/projected/805181c8-c83e-433d-9980-62f629c63f0d-kube-api-access-7q5j8\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.461785 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gsc\" (UniqueName: \"kubernetes.io/projected/8039f97e-ddca-4960-80a9-b53c24950ca9-kube-api-access-68gsc\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.461785 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461722 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-config\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.461785 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkcmc\" (UniqueName: \"kubernetes.io/projected/af42539c-c267-4894-a291-764cbb70c30f-kube-api-access-nkcmc\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.461785 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7txw\" (UniqueName: \"kubernetes.io/projected/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-kube-api-access-l7txw\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.461960 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.461960 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.461960 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461954 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.462046 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.461985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.462046 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462009 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-tmp\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.462046 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:05.462168 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.462168 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.462168 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/805181c8-c83e-433d-9980-62f629c63f0d-serving-cert\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.462249 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462207 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/20fefbf1-a4ae-4208-a927-2dc47132a88c-kube-api-access-vvjdz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.462280 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8csv\" (UniqueName: \"kubernetes.io/projected/96548fcd-4945-49fd-a9d4-1577c47e0ed1-kube-api-access-g8csv\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.462318 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462303 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-stats-auth\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.462376 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462351 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.462404 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462394 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.462442 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-snapshots\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.462522 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462502 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.462554 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.462554 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.462618 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462563 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.462654 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.462620 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.465477 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.465461 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/805181c8-c83e-433d-9980-62f629c63f0d-serving-cert\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.486722 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.486695 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5j8\" (UniqueName: \"kubernetes.io/projected/805181c8-c83e-433d-9980-62f629c63f0d-kube-api-access-7q5j8\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.486807 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.486732 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfzg\" (UniqueName: \"kubernetes.io/projected/35dff775-deae-46eb-a3bc-b5618173818f-kube-api-access-vbfzg\") pod \"kas-connection-checker-685895c7c9-xzc9s\" (UID: \"35dff775-deae-46eb-a3bc-b5618173818f\") " pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:05.563001 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.562967 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-tmp\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.563001 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563025 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/20fefbf1-a4ae-4208-a927-2dc47132a88c-kube-api-access-vvjdz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8csv\" (UniqueName: \"kubernetes.io/projected/96548fcd-4945-49fd-a9d4-1577c47e0ed1-kube-api-access-g8csv\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-stats-auth\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.563146 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563166 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.563222 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.563208 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.063190991 +0000 UTC m=+31.667036570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.563238 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.563269 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.063259879 +0000 UTC m=+31.667105448 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-snapshots\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563417 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563448 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-tmp\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563492 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.563524 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.563579 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.06356318 +0000 UTC m=+31.667408763 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:05.563595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7qq\" (UniqueName: \"kubernetes.io/projected/92342503-edc9-45e9-b9d7-fff71ce5092d-kube-api-access-9s7qq\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-service-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563696 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8039f97e-ddca-4960-80a9-b53c24950ca9-tmp-dir\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563729 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563737 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8039f97e-ddca-4960-80a9-b53c24950ca9-config-volume\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.563760 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564010 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-snapshots\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.564033 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.564049 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-default-certificate\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564115 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brsk7\" (UniqueName: \"kubernetes.io/projected/a36d3876-b5ff-48cd-913c-48108ed86f77-kube-api-access-brsk7\") pod \"network-check-source-6f78d59f99-ll9q5\" (UID: \"a36d3876-b5ff-48cd-913c-48108ed86f77\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:05.564186 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564164 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.564782 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564196 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.564782 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564302 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8039f97e-ddca-4960-80a9-b53c24950ca9-tmp-dir\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.564782 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.564452 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.064434581 +0000 UTC m=+31.668280146 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:05.564782 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8039f97e-ddca-4960-80a9-b53c24950ca9-config-volume\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.564996 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-service-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.564996 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92342503-edc9-45e9-b9d7-fff71ce5092d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.564996 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.564971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20fefbf1-a4ae-4208-a927-2dc47132a88c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.565130 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565001 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.565130 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.565130 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjv9\" (UniqueName: \"kubernetes.io/projected/8f52e908-ef31-4672-ac4d-1ee9778d1b52-kube-api-access-9wjv9\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.565130 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz678\" (UniqueName: \"kubernetes.io/projected/b67627fb-3093-4fff-9ab1-c0a4432f72e2-kube-api-access-lz678\") pod \"kas-connection-checker-685895c7c9-v5stm\" (UID: \"b67627fb-3093-4fff-9ab1-c0a4432f72e2\") " pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565128 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71bf711-37d2-421a-8bb5-a838f60d36dc-cert\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndmz\" (UniqueName: \"kubernetes.io/projected/5abb2415-7417-4478-9d12-a4d9f2e0549c-kube-api-access-jndmz\") pod \"volume-data-source-validator-58c9f5f46d-cqqrp\" (UID: \"5abb2415-7417-4478-9d12-a4d9f2e0549c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f52e908-ef31-4672-ac4d-1ee9778d1b52-serving-cert\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20fefbf1-a4ae-4208-a927-2dc47132a88c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565271 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565302 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565314 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.565342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565328 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs699\" (UniqueName: \"kubernetes.io/projected/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e-kube-api-access-hs699\") pod \"kas-connection-checker-685895c7c9-6sww9\" (UID: \"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e\") " pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2jl\" (UniqueName: \"kubernetes.io/projected/e71bf711-37d2-421a-8bb5-a838f60d36dc-kube-api-access-4r2jl\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565436 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565460 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565488 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fec1a19-aa94-423d-a573-71c8b0739f55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565528 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gsc\" (UniqueName: \"kubernetes.io/projected/8039f97e-ddca-4960-80a9-b53c24950ca9-kube-api-access-68gsc\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-config\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565582 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkcmc\" (UniqueName: \"kubernetes.io/projected/af42539c-c267-4894-a291-764cbb70c30f-kube-api-access-nkcmc\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7txw\" (UniqueName: \"kubernetes.io/projected/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-kube-api-access-l7txw\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565486 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.565746 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.565749 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.065728191 +0000 UTC m=+31.669573778 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:05.566402 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.565959 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.566455 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.566440 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fec1a19-aa94-423d-a573-71c8b0739f55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:05.566646 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.566531 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:05.566646 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.566578 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.066563736 +0000 UTC m=+31.670409304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:05.566646 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.566584 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-config\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.566646 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.566532 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.566898 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.566665 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:05.566898 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.566671 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-default-certificate\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.566898 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.566696 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.066685326 +0000 UTC m=+31.670530896 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:05.567179 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567151 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-stats-auth\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.567426 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567402 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.567544 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567487 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92342503-edc9-45e9-b9d7-fff71ce5092d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.567696 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567669 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20fefbf1-a4ae-4208-a927-2dc47132a88c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.567910 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567884 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.568003 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567928 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.568003 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.567955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.568109 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.568057 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71bf711-37d2-421a-8bb5-a838f60d36dc-cert\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.568490 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.568466 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.568914 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.568894 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.569085 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.569070 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:05.569200 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.569183 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:05.569371 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.569293 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.569371 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.569100 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.569371 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.569341 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.569613 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:05.569392 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:06.069325047 +0000 UTC m=+31.673170629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:05.570043 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.570025 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f52e908-ef31-4672-ac4d-1ee9778d1b52-serving-cert\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.570141 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.570094 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.570601 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.570580 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20fefbf1-a4ae-4208-a927-2dc47132a88c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.571032 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.571014 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.590400 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.590379 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:05.597500 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.597482 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8csv\" (UniqueName: \"kubernetes.io/projected/96548fcd-4945-49fd-a9d4-1577c47e0ed1-kube-api-access-g8csv\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:05.600151 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.600127 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:05.600297 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.600243 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7qq\" (UniqueName: \"kubernetes.io/projected/92342503-edc9-45e9-b9d7-fff71ce5092d-kube-api-access-9s7qq\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:05.601335 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.601006 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.601335 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.601032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs699\" (UniqueName: \"kubernetes.io/projected/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e-kube-api-access-hs699\") pod \"kas-connection-checker-685895c7c9-6sww9\" (UID: \"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e\") " pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:05.601335 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.601279 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndmz\" (UniqueName: \"kubernetes.io/projected/5abb2415-7417-4478-9d12-a4d9f2e0549c-kube-api-access-jndmz\") pod \"volume-data-source-validator-58c9f5f46d-cqqrp\" (UID: \"5abb2415-7417-4478-9d12-a4d9f2e0549c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:05.601507 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.601381 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.601507 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.601478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/20fefbf1-a4ae-4208-a927-2dc47132a88c-kube-api-access-vvjdz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.601606 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.601515 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:05.603091 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.603036 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brsk7\" (UniqueName: \"kubernetes.io/projected/a36d3876-b5ff-48cd-913c-48108ed86f77-kube-api-access-brsk7\") pod \"network-check-source-6f78d59f99-ll9q5\" (UID: \"a36d3876-b5ff-48cd-913c-48108ed86f77\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:05.603196 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.603097 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:05.603579 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.603556 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjv9\" (UniqueName: \"kubernetes.io/projected/8f52e908-ef31-4672-ac4d-1ee9778d1b52-kube-api-access-9wjv9\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.603772 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.603751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2jl\" (UniqueName: \"kubernetes.io/projected/e71bf711-37d2-421a-8bb5-a838f60d36dc-kube-api-access-4r2jl\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.604484 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.604451 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkcmc\" (UniqueName: \"kubernetes.io/projected/af42539c-c267-4894-a291-764cbb70c30f-kube-api-access-nkcmc\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:05.604605 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.604581 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gsc\" (UniqueName: \"kubernetes.io/projected/8039f97e-ddca-4960-80a9-b53c24950ca9-kube-api-access-68gsc\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:05.605086 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.605065 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz678\" (UniqueName: \"kubernetes.io/projected/b67627fb-3093-4fff-9ab1-c0a4432f72e2-kube-api-access-lz678\") pod \"kas-connection-checker-685895c7c9-v5stm\" (UID: \"b67627fb-3093-4fff-9ab1-c0a4432f72e2\") " pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:05.605384 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.605367 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7txw\" (UniqueName: \"kubernetes.io/projected/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-kube-api-access-l7txw\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.620983 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.620957 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:05.632558 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.632529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:05.634095 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.634068 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:05.656402 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.656372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:05.683632 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.683058 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:05.684566 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.684207 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:05.697314 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.696514 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:05.715657 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.707845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:05.715657 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.709283 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:05.888431 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.883737 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:05.888431 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.884772 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:05.908457 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.908253 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w4rqx\"" Oct 31 07:29:05.908457 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.908331 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 07:29:05.908457 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.908432 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lzwlj\"" Oct 31 07:29:05.915647 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.915201 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-xzc9s"] Oct 31 07:29:05.920075 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.919987 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk"] Oct 31 07:29:05.920185 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:05.920072 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dff775_deae_46eb_a3bc_b5618173818f.slice/crio-26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461 WatchSource:0}: Error finding container 26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461: Status 404 returned error can't find the container with id 26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461 Oct 31 07:29:05.921369 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:05.921347 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod805181c8_c83e_433d_9980_62f629c63f0d.slice/crio-fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234 WatchSource:0}: Error finding container fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234: Status 404 returned error can't find the container with id fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234 Oct 31 07:29:05.954466 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.954431 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6sww9"] Oct 31 07:29:05.955130 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:05.955102 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d0e8e1_a3d4_4a15_a9bc_17cde0e5df6e.slice/crio-0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a WatchSource:0}: Error finding container 0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a: Status 404 returned error can't find the container with id 0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a Oct 31 07:29:05.958746 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:05.958720 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-v5stm"] Oct 31 07:29:05.959509 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:05.959350 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67627fb_3093_4fff_9ab1_c0a4432f72e2.slice/crio-f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e WatchSource:0}: Error finding container f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e: Status 404 returned error can't find the container with id f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e Oct 31 07:29:06.016733 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.015160 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-vvm6c"] Oct 31 07:29:06.020564 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:06.020519 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f52e908_ef31_4672_ac4d_1ee9778d1b52.slice/crio-93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021 WatchSource:0}: Error finding container 93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021: Status 404 returned error can't find the container with id 93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021 Oct 31 07:29:06.055558 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.055409 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj"] Oct 31 07:29:06.059020 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.058995 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5"] Oct 31 07:29:06.061412 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:06.061383 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fefbf1_a4ae_4208_a927_2dc47132a88c.slice/crio-148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21 WatchSource:0}: Error finding container 148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21: Status 404 returned error can't find the container with id 148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21 Oct 31 07:29:06.061743 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:06.061718 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda36d3876_b5ff_48cd_913c_48108ed86f77.slice/crio-89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b WatchSource:0}: Error finding container 89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b: Status 404 returned error can't find the container with id 89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b Oct 31 07:29:06.069548 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:06.069525 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5abb2415_7417_4478_9d12_a4d9f2e0549c.slice/crio-5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8 WatchSource:0}: Error finding container 5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8: Status 404 returned error can't find the container with id 5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8 Oct 31 07:29:06.070408 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.070266 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp"] Oct 31 07:29:06.088305 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088286 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:06.088377 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088322 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:06.088377 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088365 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:06.088474 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088428 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:06.088521 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088484 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088465538 +0000 UTC m=+32.692311117 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:06.088521 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088433 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:06.088521 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088502 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:06.088521 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088508 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:06.088521 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088517 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088503 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088552 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088537686 +0000 UTC m=+32.692383270 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088571 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088559291 +0000 UTC m=+32.692404880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088583 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088577564 +0000 UTC m=+32.692423128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088625 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088650 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:06.088706 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.088687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088730 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088764 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088753581 +0000 UTC m=+32.692599159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088810 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088846 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088835592 +0000 UTC m=+32.692681170 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088855 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088862 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088852827 +0000 UTC m=+32.692698394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088865 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:06.089048 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:06.088889 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:07.088882991 +0000 UTC m=+32.692728558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:06.141681 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:06.141648 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a619c0_1d45_4bfe_b948_f79f3d461cb2.slice/crio-6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35 WatchSource:0}: Error finding container 6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35: Status 404 returned error can't find the container with id 6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35 Oct 31 07:29:06.142468 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.142448 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-hzng6"] Oct 31 07:29:06.150951 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:06.150924 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode71bf711_37d2_421a_8bb5_a838f60d36dc.slice/crio-787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589 WatchSource:0}: Error finding container 787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589: Status 404 returned error can't find the container with id 787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589 Oct 31 07:29:06.151442 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.151424 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-g29bz"] Oct 31 07:29:06.220827 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.220735 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" event={"ID":"8f52e908-ef31-4672-ac4d-1ee9778d1b52","Type":"ContainerStarted","Data":"93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021"} Oct 31 07:29:06.221718 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.221688 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" event={"ID":"a4a619c0-1d45-4bfe-b948-f79f3d461cb2","Type":"ContainerStarted","Data":"6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35"} Oct 31 07:29:06.223189 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.223161 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v5stm" event={"ID":"b67627fb-3093-4fff-9ab1-c0a4432f72e2","Type":"ContainerStarted","Data":"84d65514928da2413c41906a0125289451426915d4121a50db9351c1f50990c2"} Oct 31 07:29:06.223292 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.223203 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-v5stm" event={"ID":"b67627fb-3093-4fff-9ab1-c0a4432f72e2","Type":"ContainerStarted","Data":"f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e"} Oct 31 07:29:06.224529 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.224505 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6sww9" event={"ID":"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e","Type":"ContainerStarted","Data":"cb64b40b0901529a4bbf40e42aec58dbe7e89dbe8f1a4f314c69e5718f4452bf"} Oct 31 07:29:06.224620 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.224543 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6sww9" event={"ID":"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e","Type":"ContainerStarted","Data":"0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a"} Oct 31 07:29:06.227648 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.225879 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" event={"ID":"805181c8-c83e-433d-9980-62f629c63f0d","Type":"ContainerStarted","Data":"fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234"} Oct 31 07:29:06.228288 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.228259 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" event={"ID":"35dff775-deae-46eb-a3bc-b5618173818f","Type":"ContainerStarted","Data":"9d9af4bfecdcd22ef06ba640cfc95b36f9ac8e0edb71460e9c0946bf2a52c9a2"} Oct 31 07:29:06.228367 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.228295 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" event={"ID":"35dff775-deae-46eb-a3bc-b5618173818f","Type":"ContainerStarted","Data":"26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461"} Oct 31 07:29:06.229807 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.229776 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" event={"ID":"5abb2415-7417-4478-9d12-a4d9f2e0549c","Type":"ContainerStarted","Data":"5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8"} Oct 31 07:29:06.230910 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.230891 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g29bz" event={"ID":"e71bf711-37d2-421a-8bb5-a838f60d36dc","Type":"ContainerStarted","Data":"787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589"} Oct 31 07:29:06.231897 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.231877 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" event={"ID":"a36d3876-b5ff-48cd-913c-48108ed86f77","Type":"ContainerStarted","Data":"89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b"} Oct 31 07:29:06.232804 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.232777 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" event={"ID":"20fefbf1-a4ae-4208-a927-2dc47132a88c","Type":"ContainerStarted","Data":"148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21"} Oct 31 07:29:06.268452 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.268408 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-v5stm" podStartSLOduration=397.268396151 podStartE2EDuration="6m37.268396151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.268153293 +0000 UTC m=+31.871998878" watchObservedRunningTime="2026-10-31 07:29:06.268396151 +0000 UTC m=+31.872241745" Oct 31 07:29:06.313820 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.313587 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6sww9" podStartSLOduration=397.313571614 podStartE2EDuration="6m37.313571614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.313354451 +0000 UTC m=+31.917200038" watchObservedRunningTime="2026-10-31 07:29:06.313571614 +0000 UTC m=+31.917417202" Oct 31 07:29:06.358326 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.358269 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" podStartSLOduration=397.358250109 podStartE2EDuration="6m37.358250109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:06.357975884 +0000 UTC m=+31.961821470" watchObservedRunningTime="2026-10-31 07:29:06.358250109 +0000 UTC m=+31.962095696" Oct 31 07:29:06.645595 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.645478 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:06.756262 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.756227 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:06.888342 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.888312 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:06.906749 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.906724 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 07:29:06.907148 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:06.906831 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tdhmc\"" Oct 31 07:29:07.100098 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100057 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:07.100251 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:07.100251 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:07.100251 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100223 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:07.100419 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:07.100419 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100299 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:07.100419 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100334 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:07.100419 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.100377 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:07.100616 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.100524 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:07.100616 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.100586 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.100566938 +0000 UTC m=+34.704412515 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101008 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101062 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.101045658 +0000 UTC m=+34.704891239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101119 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101166 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.101142452 +0000 UTC m=+34.704988032 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101233 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101245 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101278 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.101267172 +0000 UTC m=+34.705112754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101330 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101361 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.101351756 +0000 UTC m=+34.705197335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101415 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101447 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.101435364 +0000 UTC m=+34.705280942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101506 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.101496764 +0000 UTC m=+34.705342344 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101561 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101570 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:07.101685 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:07.101597 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:09.10158728 +0000 UTC m=+34.705432861 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:07.259562 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.259486 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:07.373512 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.373474 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:07.687221 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.687187 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:07.800854 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:07.800732 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:08.718698 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.718659 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:08.719259 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.718761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:08.719259 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:08.719036 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 07:29:08.719259 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:08.719105 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:40.719087377 +0000 UTC m=+66.322932964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : secret "metrics-daemon-secret" not found Oct 31 07:29:08.724473 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.724420 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:08.725402 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.725216 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:08.820204 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.819575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:08.844365 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.844304 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:08.927051 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:08.927019 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:09.072735 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.072704 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-l5c85"] Oct 31 07:29:09.077907 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:09.077868 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c31ebf7_17d4_4aef_824b_afd78bd52d6e.slice/crio-e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4 WatchSource:0}: Error finding container e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4: Status 404 returned error can't find the container with id e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4 Oct 31 07:29:09.123103 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:09.123244 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:09.123244 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:09.123244 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123214 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:09.123415 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:09.123415 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123292 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:09.123415 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:09.123415 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.123360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:09.123606 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.123537 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:09.123606 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.123553 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:09.123834 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.123608 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.123589747 +0000 UTC m=+38.727435329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:09.124026 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124006 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:09.124094 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124063 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124049681 +0000 UTC m=+38.727895267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:09.124208 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124129 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:09.124208 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124164 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124153099 +0000 UTC m=+38.727998682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:09.124308 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124228 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124218912 +0000 UTC m=+38.728064481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:09.124308 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124290 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:09.124308 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124301 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124329 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124319545 +0000 UTC m=+38.728165122 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124386 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124416 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124406734 +0000 UTC m=+38.728252315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124474 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124500 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124491405 +0000 UTC m=+38.728337000 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124555 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:09.124627 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:09.124584 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.124572859 +0000 UTC m=+38.728418437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:09.252864 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.252828 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" event={"ID":"5abb2415-7417-4478-9d12-a4d9f2e0549c","Type":"ContainerStarted","Data":"25596154603a4fb3925f84a525f8e6b65434155c7b53e0780606541fc3ac3f77"} Oct 31 07:29:09.255051 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.254582 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5c85" event={"ID":"2c31ebf7-17d4-4aef-824b-afd78bd52d6e","Type":"ContainerStarted","Data":"e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4"} Oct 31 07:29:09.255278 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.255258 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gdgxp"] Oct 31 07:29:09.258357 ip-10-0-140-6 kubenswrapper[2580]: W1031 07:29:09.258320 2580 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b34ebf_4617_4662_ad56_5dbbb97a4e68.slice/crio-b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4 WatchSource:0}: Error finding container b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4: Status 404 returned error can't find the container with id b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4 Oct 31 07:29:09.308164 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.308098 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" podStartSLOduration=371.167866429 podStartE2EDuration="6m13.308078693s" totalImagesPullingTime="2.140212264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:09.297280897 +0000 UTC m=+34.901126486" watchObservedRunningTime="2026-10-31 07:29:09.308078693 +0000 UTC m=+34.911924284" Oct 31 07:29:09.416838 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.416717 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:09.526191 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:09.526011 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:10.264489 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:10.264451 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gdgxp" event={"ID":"a6b34ebf-4617-4662-ad56-5dbbb97a4e68","Type":"ContainerStarted","Data":"b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4"} Oct 31 07:29:11.276274 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.275524 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" event={"ID":"805181c8-c83e-433d-9980-62f629c63f0d","Type":"ContainerStarted","Data":"b32826ef5a483d8eb48dab8d6a9a8fc08deb090a89970f8cf2a4eea407f8acaf"} Oct 31 07:29:11.279429 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.278983 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" event={"ID":"20fefbf1-a4ae-4208-a927-2dc47132a88c","Type":"ContainerStarted","Data":"50f7cd999db68fd915663a71487b9e05e7287bc9699ed12821c54f7f32a3bc3c"} Oct 31 07:29:11.281602 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.281211 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" event={"ID":"8f52e908-ef31-4672-ac4d-1ee9778d1b52","Type":"ContainerStarted","Data":"405d214e96a6aee2a8dac5e60c14440585b71672dabb1c37fe067678c1f2b127"} Oct 31 07:29:11.284257 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.283736 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/0.log" Oct 31 07:29:11.284257 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.283760 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" containerID="826fb2d112be86574542fd3b04201d638c172cef2a89fe06884db4c58f169692" exitCode=255 Oct 31 07:29:11.284257 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.283779 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" event={"ID":"a4a619c0-1d45-4bfe-b948-f79f3d461cb2","Type":"ContainerDied","Data":"826fb2d112be86574542fd3b04201d638c172cef2a89fe06884db4c58f169692"} Oct 31 07:29:11.284257 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.284028 2580 scope.go:122] "RemoveContainer" containerID="826fb2d112be86574542fd3b04201d638c172cef2a89fe06884db4c58f169692" Oct 31 07:29:11.321988 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.321938 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" podStartSLOduration=370.528031874 podStartE2EDuration="6m15.321920612s" totalImagesPullingTime="4.793888738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:11.320980461 +0000 UTC m=+36.924826046" watchObservedRunningTime="2026-10-31 07:29:11.321920612 +0000 UTC m=+36.925766199" Oct 31 07:29:11.418193 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.418134 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" podStartSLOduration=370.719792141 podStartE2EDuration="6m15.418116272s" totalImagesPullingTime="4.698324131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:11.417480906 +0000 UTC m=+37.021326504" watchObservedRunningTime="2026-10-31 07:29:11.418116272 +0000 UTC m=+37.021961860" Oct 31 07:29:11.464690 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.464608 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" podStartSLOduration=370.449359459 podStartE2EDuration="6m15.464590785s" totalImagesPullingTime="5.015231326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:11.463226304 +0000 UTC m=+37.067071903" watchObservedRunningTime="2026-10-31 07:29:11.464590785 +0000 UTC m=+37.068436371" Oct 31 07:29:11.861998 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.860739 2580 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zdg8b"] Oct 31 07:29:11.884376 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.884272 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:11.888687 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.888600 2580 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdg8b"] Oct 31 07:29:11.918650 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.918591 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-48xxx\"" Oct 31 07:29:11.918921 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.918880 2580 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 07:29:11.938038 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.936567 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:11.955820 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.955684 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:11.955820 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.955752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e2c4370-aeba-4e68-ad8d-396664e91a83-data-volume\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:11.956017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.955824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsxxh\" (UniqueName: \"kubernetes.io/projected/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-api-access-dsxxh\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:11.956017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.955919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:11.956017 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.956002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e2c4370-aeba-4e68-ad8d-396664e91a83-crio-socket\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:11.994532 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:11.994443 2580 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 07:29:12.042523 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.042494 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.057169 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxxh\" (UniqueName: \"kubernetes.io/projected/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-api-access-dsxxh\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.057291 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.057387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e2c4370-aeba-4e68-ad8d-396664e91a83-crio-socket\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.057457 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.057533 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.057563 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.057599 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:12.557581281 +0000 UTC m=+38.161426861 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : configmap "kube-rbac-proxy" not found Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.057623 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:12.557607169 +0000 UTC m=+38.161452738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.057559 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e2c4370-aeba-4e68-ad8d-396664e91a83-crio-socket\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.057670 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.057669 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e2c4370-aeba-4e68-ad8d-396664e91a83-data-volume\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.058264 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.058032 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e2c4370-aeba-4e68-ad8d-396664e91a83-data-volume\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.085030 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.085001 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxxh\" (UniqueName: \"kubernetes.io/projected/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-api-access-dsxxh\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.231150 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.231116 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:12.291965 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.291936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/1.log" Oct 31 07:29:12.292391 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.292373 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/0.log" Oct 31 07:29:12.292445 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.292412 2580 generic.go:356] "Generic (PLEG): container finished" podID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" containerID="0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4" exitCode=255 Oct 31 07:29:12.292555 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.292537 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" event={"ID":"a4a619c0-1d45-4bfe-b948-f79f3d461cb2","Type":"ContainerDied","Data":"0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4"} Oct 31 07:29:12.292603 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.292591 2580 scope.go:122] "RemoveContainer" containerID="826fb2d112be86574542fd3b04201d638c172cef2a89fe06884db4c58f169692" Oct 31 07:29:12.292942 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.292929 2580 scope.go:122] "RemoveContainer" containerID="0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4" Oct 31 07:29:12.293116 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.293101 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hzng6_openshift-console-operator(a4a619c0-1d45-4bfe-b948-f79f3d461cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" podUID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" Oct 31 07:29:12.295877 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.295850 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-g29bz" event={"ID":"e71bf711-37d2-421a-8bb5-a838f60d36dc","Type":"ContainerStarted","Data":"90fd98c410e4c00885ca03fd69cc63e3dcff97bd6207f6292b3343a3b4360017"} Oct 31 07:29:12.358996 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.358925 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:12.394507 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.394464 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-g29bz" podStartSLOduration=1.868682589 podStartE2EDuration="7.394451024s" totalImagesPullingTime="5.525768435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:12.39416561 +0000 UTC m=+37.998011196" watchObservedRunningTime="2026-10-31 07:29:12.394451024 +0000 UTC m=+37.998296609" Oct 31 07:29:12.466081 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.466045 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:12.562631 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.562548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.562777 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.562702 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:12.562777 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.562717 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:12.562919 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.562826 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.56278108 +0000 UTC m=+39.166626651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:12.562919 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.562857 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 07:29:12.562919 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:12.562909 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:13.562894339 +0000 UTC m=+39.166739917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : configmap "kube-rbac-proxy" not found Oct 31 07:29:12.743312 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.743276 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:12.866270 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:12.866200 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:13.168631 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168551 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:13.168631 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:13.168631 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168614 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168667 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168670 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168718 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168737 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168736 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168809 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168811 2580 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168820 2580 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168742 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.168721311 +0000 UTC m=+46.772566893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168895 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.168878141 +0000 UTC m=+46.772723706 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168909 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.168902147 +0000 UTC m=+46.772747710 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:13.168925 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168925 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.168919692 +0000 UTC m=+46.772765256 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168942 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.168936991 +0000 UTC m=+46.772782554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.168956 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.168951344 +0000 UTC m=+46.772796907 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.168987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.169022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.169107 2580 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.169107 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.169124 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.169135 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.169128744 +0000 UTC m=+46.772974307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:13.169375 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.169159 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:21.16914703 +0000 UTC m=+46.772992598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:13.258211 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.258180 2580 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:13.300734 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.300697 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" event={"ID":"a36d3876-b5ff-48cd-913c-48108ed86f77","Type":"ContainerStarted","Data":"96e735f491c777889fbf3f4e73917d83af9b3336db485d074ed6a67202170bc4"} Oct 31 07:29:13.302759 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.302739 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/1.log" Oct 31 07:29:13.303056 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.303039 2580 scope.go:122] "RemoveContainer" containerID="0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4" Oct 31 07:29:13.303263 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.303244 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hzng6_openshift-console-operator(a4a619c0-1d45-4bfe-b948-f79f3d461cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" podUID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" Oct 31 07:29:13.304618 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.304581 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gdgxp" event={"ID":"a6b34ebf-4617-4662-ad56-5dbbb97a4e68","Type":"ContainerStarted","Data":"6c3f43b0c4adc00e6d3324a459b94380d347fe8e052f00c601a679580dd4ab75"} Oct 31 07:29:13.350830 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.350766 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" podStartSLOduration=368.092138041 podStartE2EDuration="6m14.350750711s" totalImagesPullingTime="6.25861267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:13.349721066 +0000 UTC m=+38.953566662" watchObservedRunningTime="2026-10-31 07:29:13.350750711 +0000 UTC m=+38.954596297" Oct 31 07:29:13.444423 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.444312 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gdgxp" podStartSLOduration=34.021687638 podStartE2EDuration="37.44429539s" totalImagesPullingTime="3.422607752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:13.443137959 +0000 UTC m=+39.046983547" watchObservedRunningTime="2026-10-31 07:29:13.44429539 +0000 UTC m=+39.048140976" Oct 31 07:29:13.573591 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.573550 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:13.573776 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:13.573685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:13.573776 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.573706 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:13.573904 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.573810 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:15.573777021 +0000 UTC m=+41.177622591 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:13.573904 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.573832 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 07:29:13.573904 ip-10-0-140-6 kubenswrapper[2580]: E1031 07:29:13.573898 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:15.573877848 +0000 UTC m=+41.177723419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : configmap "kube-rbac-proxy" not found Oct 31 07:29:14.308500 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:14.308467 2580 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:15.312526 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:15.312492 2580 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-l5c85" event={"ID":"2c31ebf7-17d4-4aef-824b-afd78bd52d6e","Type":"ContainerStarted","Data":"8566c3678ee85865b99bb55a2ba6ac14db9b2f1d4c1a371a6005a9aa6b37bb20"} Oct 31 07:29:15.358268 ip-10-0-140-6 kubenswrapper[2580]: I1031 07:29:15.358217 2580 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-l5c85" podStartSLOduration=33.269323796 podStartE2EDuration="39.358204673s" totalImagesPullingTime="6.088880877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:28:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:15.357663083 +0000 UTC m=+40.961508670" watchObservedRunningTime="2026-10-31 07:29:15.358204673 +0000 UTC m=+40.962050256" Oct 31 07:29:15.477158 ip-10-0-140-6 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 07:29:15.485366 ip-10-0-140-6 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 07:29:15.485534 ip-10-0-140-6 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 07:29:15.485582 ip-10-0-140-6 systemd[1]: kubelet.service: Consumed 2.869s CPU time. Oct 31 07:29:15.497268 ip-10-0-140-6 systemd[1]: Starting Kubernetes Kubelet... Oct 31 07:29:15.575938 ip-10-0-140-6 kubenswrapper[4870]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 07:29:15.575938 ip-10-0-140-6 kubenswrapper[4870]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 07:29:15.575938 ip-10-0-140-6 kubenswrapper[4870]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 07:29:15.575938 ip-10-0-140-6 kubenswrapper[4870]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 07:29:15.578173 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578089 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.578173 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578167 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.578173 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578173 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.578173 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578177 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.578173 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578181 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578186 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578190 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578194 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578198 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578202 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578206 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578210 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578214 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578218 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578222 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578226 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578230 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578233 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578237 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578240 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578245 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578250 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578254 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578262 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.578502 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578266 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578270 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578274 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578279 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578283 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578287 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578291 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578295 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578299 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578303 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578307 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578311 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578318 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578328 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578334 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578339 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578344 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578349 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578353 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.579385 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578357 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578362 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578366 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578370 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578375 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578379 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578383 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578388 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578393 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578397 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578401 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578407 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578413 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578418 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578422 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578427 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578431 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578436 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578440 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578445 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.580240 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578449 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578453 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578458 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578463 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578467 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578472 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578476 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578480 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578484 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578490 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578495 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578499 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578503 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578508 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578512 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578516 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578521 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578527 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578533 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.581125 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578539 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578543 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578547 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578552 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578556 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578562 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578567 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578571 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578575 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578580 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578584 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578589 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578593 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578598 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578602 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578606 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578610 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578615 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578619 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578623 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.581978 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578627 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578632 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578636 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.578642 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579895 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579906 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579911 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579916 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579920 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579924 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579928 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579933 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579937 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579941 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579945 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579950 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579954 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579958 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579963 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.582859 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579967 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579971 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579975 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579979 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579983 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579988 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579993 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.579997 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580002 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580006 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580010 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580014 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580019 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580023 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580027 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580032 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580037 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580042 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580046 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580050 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.583717 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580054 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580059 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580065 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580072 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580078 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580082 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580087 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580091 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580096 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580101 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580105 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580109 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580113 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580117 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580122 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580126 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580131 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580135 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580139 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580143 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.584563 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580148 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580152 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580157 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580162 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580167 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580171 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580175 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580180 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580184 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580188 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580193 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580198 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580202 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580207 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580211 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580215 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580220 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580224 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580228 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.585123 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580236 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580242 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580246 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580250 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580254 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580258 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580262 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580267 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580271 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580276 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580281 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580285 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580290 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580294 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580298 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580305 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580309 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580314 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580318 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580323 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.585705 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580327 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580332 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580336 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580340 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580346 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580350 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580354 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580358 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580362 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580367 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580373 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.580377 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581236 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581246 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581251 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581256 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581260 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581264 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581269 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581273 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.586293 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581278 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581282 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581287 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581291 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581296 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581300 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581304 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581314 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581320 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581325 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581329 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581334 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581338 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581343 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581347 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581351 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581363 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581368 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581374 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.586811 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581380 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581385 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581391 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581395 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581401 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581406 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581412 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581417 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581421 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581425 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581430 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581434 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581438 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581442 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581446 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581451 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581455 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581459 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581464 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.587287 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581468 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581477 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581481 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581486 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581491 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581495 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581499 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581504 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581508 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581512 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581517 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581522 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581526 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581530 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581534 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581539 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581543 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581550 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581555 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.587744 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581559 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581563 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581567 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581572 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581576 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581580 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581584 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581589 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581593 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581598 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581602 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581606 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581610 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581615 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581621 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581625 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581629 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581634 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581638 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581642 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.588214 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581647 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581651 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581655 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581659 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581663 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581668 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581673 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581677 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581681 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581686 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581692 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581698 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581702 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581706 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581710 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581714 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581719 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581723 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581727 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.588707 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581731 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.581736 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581888 4870 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581900 4870 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581910 4870 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581917 4870 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581924 4870 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581931 4870 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581938 4870 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581945 4870 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581950 4870 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581956 4870 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581962 4870 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581967 4870 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581972 4870 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581977 4870 flags.go:64] FLAG: --cgroup-root="" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581981 4870 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581987 4870 flags.go:64] FLAG: --client-ca-file="" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581991 4870 flags.go:64] FLAG: --cloud-provider="external" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.581997 4870 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582002 4870 flags.go:64] FLAG: --cluster-domain="" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582007 4870 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582013 4870 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582020 4870 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 07:29:15.589496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582026 4870 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582032 4870 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582037 4870 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582042 4870 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582047 4870 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582052 4870 flags.go:64] FLAG: --contention-profiling="false" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582057 4870 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582062 4870 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582067 4870 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582072 4870 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582079 4870 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582084 4870 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582089 4870 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582094 4870 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582099 4870 flags.go:64] FLAG: --enable-server="true" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582105 4870 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582113 4870 flags.go:64] FLAG: --event-burst="100" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582118 4870 flags.go:64] FLAG: --event-qps="50" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582123 4870 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582128 4870 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582133 4870 flags.go:64] FLAG: --eviction-hard="" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582140 4870 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582145 4870 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582150 4870 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582155 4870 flags.go:64] FLAG: --eviction-soft="" Oct 31 07:29:15.590172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582160 4870 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582165 4870 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582170 4870 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582175 4870 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582180 4870 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582185 4870 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582192 4870 flags.go:64] FLAG: --feature-gates="" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582199 4870 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582203 4870 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582208 4870 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582214 4870 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582218 4870 flags.go:64] FLAG: --healthz-port="10248" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582223 4870 flags.go:64] FLAG: --help="false" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582228 4870 flags.go:64] FLAG: --hostname-override="ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582233 4870 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582239 4870 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582244 4870 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582249 4870 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582255 4870 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582260 4870 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582265 4870 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582269 4870 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582275 4870 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 07:29:15.590812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582281 4870 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582286 4870 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582291 4870 flags.go:64] FLAG: --kube-reserved="" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582295 4870 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582300 4870 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582305 4870 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582309 4870 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582314 4870 flags.go:64] FLAG: --lock-file="" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582319 4870 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582324 4870 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582329 4870 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582338 4870 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582343 4870 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582348 4870 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582353 4870 flags.go:64] FLAG: --logging-format="text" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582360 4870 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582366 4870 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582371 4870 flags.go:64] FLAG: --manifest-url="" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582376 4870 flags.go:64] FLAG: --manifest-url-header="" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582383 4870 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582388 4870 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582394 4870 flags.go:64] FLAG: --max-pods="110" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582399 4870 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582404 4870 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582409 4870 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 07:29:15.591391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582414 4870 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582419 4870 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582423 4870 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582428 4870 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582439 4870 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582445 4870 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582451 4870 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582456 4870 flags.go:64] FLAG: --pod-cidr="" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582460 4870 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582465 4870 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582470 4870 flags.go:64] FLAG: --pods-per-core="0" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582475 4870 flags.go:64] FLAG: --port="10250" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582480 4870 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582485 4870 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0564f5668d95865ef" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582490 4870 flags.go:64] FLAG: --qos-reserved="" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582495 4870 flags.go:64] FLAG: --read-only-port="10255" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582500 4870 flags.go:64] FLAG: --register-node="true" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582505 4870 flags.go:64] FLAG: --register-with-taints="" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582511 4870 flags.go:64] FLAG: --registry-burst="10" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582515 4870 flags.go:64] FLAG: --registry-qps="5" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582520 4870 flags.go:64] FLAG: --reserved-cpus="" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582526 4870 flags.go:64] FLAG: --reserved-memory="" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582534 4870 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582539 4870 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582544 4870 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582548 4870 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 07:29:15.592031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582553 4870 flags.go:64] FLAG: --runonce="false" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582567 4870 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582572 4870 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582577 4870 flags.go:64] FLAG: --seccomp-default="false" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582582 4870 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582587 4870 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582592 4870 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582597 4870 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582602 4870 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582607 4870 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582612 4870 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582616 4870 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582624 4870 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582629 4870 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582633 4870 flags.go:64] FLAG: --system-cgroups="" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582638 4870 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582648 4870 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582653 4870 flags.go:64] FLAG: --tls-cert-file="" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582657 4870 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582663 4870 flags.go:64] FLAG: --tls-min-version="" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582668 4870 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582673 4870 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582677 4870 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582682 4870 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582687 4870 flags.go:64] FLAG: --v="2" Oct 31 07:29:15.592667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582694 4870 flags.go:64] FLAG: --version="false" Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582700 4870 flags.go:64] FLAG: --vmodule="" Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582706 4870 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.582722 4870 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.582977 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.582985 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.582992 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.582998 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583003 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583007 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583012 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583016 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583021 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583025 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583030 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583035 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583039 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583043 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583049 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583053 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.593333 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583058 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583062 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583067 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583071 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583075 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583080 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583084 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583091 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583097 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583102 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583107 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583111 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583115 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583120 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583124 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583134 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583141 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583146 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583150 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583155 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.593855 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583159 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583164 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583168 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583172 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583177 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583181 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583185 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583190 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583194 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583198 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583202 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583207 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583212 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583216 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583220 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583224 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583229 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583233 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583238 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.594384 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583242 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583246 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583251 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583255 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583259 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583264 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583268 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583272 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583279 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583283 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583289 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583293 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583297 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583302 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583306 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583310 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583314 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583318 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583323 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583327 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.595004 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583332 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583336 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583340 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583344 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583348 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583353 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583357 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583361 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583366 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583370 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583375 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583379 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583384 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583388 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583392 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583396 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583401 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583405 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583409 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583414 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.595559 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583421 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583425 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583430 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583435 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583439 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583443 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583447 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583451 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583455 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583460 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583465 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.583506 4870 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583566 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583574 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583579 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.596067 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583583 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583588 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583592 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583596 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583601 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583605 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583612 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583616 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583621 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583625 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583629 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583634 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583638 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583642 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583647 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583651 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583656 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583663 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583669 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.596512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583676 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583682 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583688 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583692 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583696 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583700 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583705 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583709 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583713 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583717 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583721 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583726 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583730 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583734 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583738 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583742 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583747 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583751 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583756 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.597070 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583762 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583769 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583774 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583780 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583785 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583807 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583812 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583818 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583823 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583827 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583832 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583838 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583843 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583847 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583851 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583855 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583861 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583866 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583871 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583875 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.597658 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583880 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583884 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583888 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583893 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583897 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583901 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583906 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583910 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583915 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583919 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583924 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583928 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583934 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583939 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583943 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583948 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583952 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583957 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583961 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.598205 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583965 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583970 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583974 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583978 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583984 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583988 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583993 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.583998 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584002 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584006 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584011 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584016 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584020 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584024 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584028 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584033 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584037 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584041 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584046 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584052 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.598656 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584056 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584060 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584064 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584069 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584073 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.584080 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.588072 4870 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.588087 4870 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.588104 4870 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.588109 4870 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588183 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588189 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588192 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588194 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588197 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588200 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588203 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588206 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588209 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.599225 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588211 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588214 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588217 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588225 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588229 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588232 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588235 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588237 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588240 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588243 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588246 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588248 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588251 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588255 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588257 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588260 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588263 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588266 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588268 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.599691 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588272 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588275 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588277 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588280 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588282 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588285 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588288 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588290 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588293 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588295 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588298 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588301 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588303 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588306 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588308 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588311 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588314 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588322 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588325 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588328 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.600271 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588331 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588333 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588336 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588339 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588341 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588344 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588347 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588350 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588352 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588355 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588357 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588360 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588362 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588365 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588367 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588370 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588373 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588375 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588378 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.600737 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588380 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588383 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588387 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588391 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588394 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588397 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588399 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588402 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588405 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588408 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588410 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588421 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588424 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588427 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588429 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588432 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588434 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588437 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588440 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588442 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.601224 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588445 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588449 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588452 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588456 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588459 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588461 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588464 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588467 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588469 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588472 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588474 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588477 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588479 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588482 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588485 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588487 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588491 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588493 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.601777 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.588496 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.588524 4870 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589495 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589524 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589529 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589539 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589569 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589574 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589578 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589582 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589587 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589598 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589602 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589606 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589610 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589614 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.602428 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589618 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589622 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589631 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589635 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589640 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589644 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589648 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589653 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589658 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.589662 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590135 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590240 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590245 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590248 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590252 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590256 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590259 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590263 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590267 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590269 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.602824 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590272 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590275 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590278 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590282 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590285 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590288 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590291 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590295 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590300 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590303 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590306 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590309 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590312 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590315 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590317 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590320 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590323 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590326 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590331 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.603307 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590334 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590337 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590339 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590342 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590345 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590347 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590350 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590353 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590356 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590358 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590361 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590363 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590367 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590369 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590372 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590375 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590379 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590382 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590385 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.603761 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590387 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590390 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590393 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590395 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590398 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590400 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590403 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590406 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590409 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590411 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590414 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590417 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590419 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590422 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590427 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590429 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590432 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590436 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590439 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590441 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.604361 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590444 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590447 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590449 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590452 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590456 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590460 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590464 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590467 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590470 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590473 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590477 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590479 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590482 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590485 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.590519 4870 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590573 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 07:29:15.604854 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590578 4870 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590581 4870 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590584 4870 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590587 4870 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590590 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590592 4870 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590595 4870 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590597 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590600 4870 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590603 4870 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590606 4870 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590608 4870 feature_gate.go:386] unrecognized feature gate: Example Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590612 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590615 4870 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590618 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590621 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590624 4870 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590627 4870 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590629 4870 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590632 4870 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 07:29:15.605347 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590635 4870 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590637 4870 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590640 4870 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590643 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590645 4870 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590648 4870 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590651 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590654 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590657 4870 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590660 4870 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590662 4870 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590665 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590669 4870 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590672 4870 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590675 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590678 4870 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590681 4870 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590683 4870 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590686 4870 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 07:29:15.605871 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590688 4870 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590693 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590696 4870 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590699 4870 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590701 4870 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590704 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590707 4870 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590709 4870 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590712 4870 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590714 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590717 4870 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590719 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590722 4870 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590726 4870 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590728 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590731 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590733 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590736 4870 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590739 4870 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590742 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 07:29:15.606402 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590745 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590747 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590750 4870 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590752 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590755 4870 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590758 4870 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590760 4870 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590762 4870 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590765 4870 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590768 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590770 4870 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590773 4870 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590775 4870 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590779 4870 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590782 4870 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590785 4870 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590801 4870 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590804 4870 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590807 4870 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590809 4870 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 07:29:15.606907 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590812 4870 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590815 4870 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590817 4870 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590820 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590823 4870 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590826 4870 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590828 4870 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590831 4870 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590835 4870 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590839 4870 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590842 4870 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590845 4870 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590850 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590853 4870 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590855 4870 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590858 4870 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590861 4870 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590863 4870 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590866 4870 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 07:29:15.607436 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590869 4870 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590871 4870 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590874 4870 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590877 4870 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590880 4870 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590882 4870 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:15.590886 4870 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.590945 4870 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.591884 4870 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.591943 4870 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.592128 4870 server.go:1014] "Starting client certificate rotation" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.592266 4870 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.592314 4870 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 07:23:35 +0000 UTC" deadline="2028-03-27 01:50:58.635767194 +0000 UTC" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.592342 4870 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12306h21m43.0434275s" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.592546 4870 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.593560 4870 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 07:29:15.608122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.594841 4870 log.go:25] "Validated CRI v1 runtime API" Oct 31 07:29:15.608737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.596447 4870 log.go:25] "Validated CRI v1 image API" Oct 31 07:29:15.608737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.597157 4870 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 07:29:15.608737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.599193 4870 fs.go:135] Filesystem UUIDs: map[17aef96a-daa3-461d-8a39-7c96daa5ad65:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 dab11254-8006-4d6d-bead-2949b25410f8:/dev/nvme0n1p4] Oct 31 07:29:15.608907 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.599212 4870 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a/userdata/shm major:0 minor:232 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/306bb57eb3ae4766b2515fc3efe62bdccc18cffe818844e2c53028c98d654a48/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/306bb57eb3ae4766b2515fc3efe62bdccc18cffe818844e2c53028c98d654a48/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/078fca68-9d2f-4df5-924c-66d9c4198984/volumes/kubernetes.io~projected/kube-api-access-p94ln:{mountpoint:/var/lib/kubelet/pods/078fca68-9d2f-4df5-924c-66d9c4198984/volumes/kubernetes.io~projected/kube-api-access-p94ln major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e2c4370-aeba-4e68-ad8d-396664e91a83/volumes/kubernetes.io~projected/kube-api-access-dsxxh:{mountpoint:/var/lib/kubelet/pods/0e2c4370-aeba-4e68-ad8d-396664e91a83/volumes/kubernetes.io~projected/kube-api-access-dsxxh major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20fefbf1-a4ae-4208-a927-2dc47132a88c/volumes/kubernetes.io~projected/kube-api-access-vvjdz:{mountpoint:/var/lib/kubelet/pods/20fefbf1-a4ae-4208-a927-2dc47132a88c/volumes/kubernetes.io~projected/kube-api-access-vvjdz major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20fefbf1-a4ae-4208-a927-2dc47132a88c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/20fefbf1-a4ae-4208-a927-2dc47132a88c/volumes/kubernetes.io~secret/serving-cert major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~projected/kube-api-access-d9jxl:{mountpoint:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~projected/kube-api-access-d9jxl major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29d86a3e-2f5d-433b-835f-56df35aa18c2/volumes/kubernetes.io~projected/kube-api-access-8rqdj:{mountpoint:/var/lib/kubelet/pods/29d86a3e-2f5d-433b-835f-56df35aa18c2/volumes/kubernetes.io~projected/kube-api-access-8rqdj major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c31ebf7-17d4-4aef-824b-afd78bd52d6e/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2c31ebf7-17d4-4aef-824b-afd78bd52d6e/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e/volumes/kubernetes.io~projected/kube-api-access-hs699:{mountpoint:/var/lib/kubelet/pods/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e/volumes/kubernetes.io~projected/kube-api-access-hs699 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35dff775-deae-46eb-a3bc-b5618173818f/volumes/kubernetes.io~projected/kube-api-access-vbfzg:{mountpoint:/var/lib/kubelet/pods/35dff775-deae-46eb-a3bc-b5618173818f/volumes/kubernetes.io~projected/kube-api-access-vbfzg major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5abb2415-7417-4478-9d12-a4d9f2e0549c/volumes/kubernetes.io~projected/kube-api-access-jndmz:{mountpoint:/var/lib/kubelet/pods/5abb2415-7417-4478-9d12-a4d9f2e0549c/volumes/kubernetes.io~projected/kube-api-access-jndmz major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~projected/kube-api-access-92hm9:{mountpoint:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~projected/kube-api-access-92hm9 major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8039f97e-ddca-4960-80a9-b53c24950ca9/volumes/kubernetes.io~projected/kube-api-access-68gsc:{mountpoint:/var/lib/kubelet/pods/8039f97e-ddca-4960-80a9-b53c24950ca9/volumes/kubernetes.io~projected/kube-api-access-68gsc major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/805181c8-c83e-433d-9980-62f629c63f0d/volumes/kubernetes.io~projected/kube-api-access-7q5j8:{mountpoint:/var/lib/kubelet/pods/805181c8-c83e-433d-9980-62f629c63f0d/volumes/kubernetes.io~projected/kube-api-access-7q5j8 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/805181c8-c83e-433d-9980-62f629c63f0d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/805181c8-c83e-433d-9980-62f629c63f0d/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f52e908-ef31-4672-ac4d-1ee9778d1b52/volumes/kubernetes.io~projected/kube-api-access-9wjv9:{mountpoint:/var/lib/kubelet/pods/8f52e908-ef31-4672-ac4d-1ee9778d1b52/volumes/kubernetes.io~projected/kube-api-access-9wjv9 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f52e908-ef31-4672-ac4d-1ee9778d1b52/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/8f52e908-ef31-4672-ac4d-1ee9778d1b52/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92342503-edc9-45e9-b9d7-fff71ce5092d/volumes/kubernetes.io~projected/kube-api-access-9s7qq:{mountpoint:/var/lib/kubelet/pods/92342503-edc9-45e9-b9d7-fff71ce5092d/volumes/kubernetes.io~projected/kube-api-access-9s7qq major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/96548fcd-4945-49fd-a9d4-1577c47e0ed1/volumes/kubernetes.io~projected/kube-api-access-g8csv:{mountpoint:/var/lib/kubelet/pods/96548fcd-4945-49fd-a9d4-1577c47e0ed1/volumes/kubernetes.io~projected/kube-api-access-g8csv major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5/volumes/kubernetes.io~projected/kube-api-access-rqcq5:{mountpoint:/var/lib/kubelet/pods/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5/volumes/kubernetes.io~projected/kube-api-access-rqcq5 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4/volumes/kubernetes.io~projected/kube-api-access-82xjk:{mountpoint:/var/lib/kubelet/pods/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4/volumes/kubernetes.io~projected/kube-api-access-82xjk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e12a075-b4ec-44e3-8811-7b19f1410513/volumes/kubernetes.io~projected/kube-api-access-8jkrd:{mountpoint:/var/lib/kubelet/pods/9e12a075-b4ec-44e3-8811-7b19f1410513/volumes/kubernetes.io~projected/kube-api-access-8jkrd major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a122982a-c398-47f1-a368-fd89db8899a7/volumes/kubernetes.io~projected/kube-api-access-l6qvs:{mountpoint:/var/lib/kubelet/pods/a122982a-c398-47f1-a368-fd89db8899a7/volumes/kubernetes.io~projected/kube-api-access-l6qvs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a36d3876-b5ff-48cd-913c-48108ed86f77/volumes/kubernetes.io~projected/kube-api-access-brsk7:{mountpoint:/var/lib/kubelet/pods/a36d3876-b5ff-48cd-913c-48108ed86f77/volumes/kubernetes.io~projected/kube-api-access-brsk7 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4a619c0-1d45-4bfe-b948-f79f3d461cb2/volumes/kubernetes.io~projected/kube-api-access-l7txw:{mountpoint:/var/lib/kubelet/pods/a4a619c0-1d45-4bfe-b948-f79f3d461cb2/volumes/kubernetes.io~projected/kube-api-access-l7txw major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4a619c0-1d45-4bfe-b948-f79f3d461cb2/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a4a619c0-1d45-4bfe-b948-f79f3d461cb2/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~projected/kube-api-access-pwwmx:{mountpoint:/var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~projected/kube-api-access-pwwmx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6b34ebf-4617-4662-ad56-5dbbb97a4e68/volumes/kubernetes.io~projected/kube-api-access-qnr8n:{mountpoint:/var/lib/kubelet/pods/a6b34ebf-4617-4662-ad56-5dbbb97a4e68/volumes/kubernetes.io~projected/kube-api-access-qnr8n major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~projected/kube-api-access-nkcmc:{mountpoint:/var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~projected/kube-api-access-nkcmc major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~secret/default-certificate major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~secret/stats-auth major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volumes/kubernetes.io~projected/kube-api-access-8rklq:{mountpoint:/var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volumes/kubernetes.io~projected/kube-api-access-8rklq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b67627fb-3093-4fff-9ab1-c0a4432f72e2/volumes/kubernetes.io~projected/kube-api-access-lz678:{mountpoint:/var/lib/kubelet/pods/b67627fb-3093-4fff-9ab1-c0a4432f72e2/volumes/kubernetes.io~projected/kube-api-access-lz678 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9dacaea-1664-495c-b164-a939a91bac87/volumes/kubernetes.io~projected/kube-api-access-hb5h7:{mountpoint:/var/lib/kubelet/pods/b9dacaea-1664-495c-b164-a939a91bac87/volumes/kubernetes.io~projected/kube-api-access-hb5h7 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e71bf711-37d2-421a-8bb5-a838f60d36dc/volumes/kubernetes.io~projected/kube-api-access-4r2jl:{mountpoint:/var/lib/kubelet/pods/e71bf711-37d2-421a-8bb5-a838f60d36dc/volumes/kubernetes.io~projected/kube-api-access-4r2jl major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e71bf711-37d2-421a-8bb5-a838f60d36dc/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e71bf711-37d2-421a-8bb5-a838f60d36dc/volumes/kubernetes.io~secret/cert major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/7c4df95c14260a74af1fda1e37264e83b1eb99f6f2283748df6259b21332e2f9/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8ce47f531178aef32320c2594c85e3e198f7c0dd81076cc62dd1193c5944eea9/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/da5e58720dcce870291e112244455f01b021947618136342d0dffef7102bcad6/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/56e3867cd5d4c43982c478012e407767f4b8ac5a397fc665466e704ee1fd3b40/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ebfbbc18637d1d66038fd903588dac7e323572a6211f9b9de21f86bfdad2b966/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/4f658c42a7f378aa4a9246b90d8eef5b2d4704afd70ba2d8b81e8bdd59bee7b7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/3b751a4c319dbaad4de106988fefee79884889f56e6229045be987c4effa811b/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/f1bdcb610b37ab418c268546730330b95285a222258f1541db179da3f15c7728/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/981266c0f70a8930db64e618831da08aeb64c3bef4bd67a41b7f20f9fcda3b81/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/163a6de0b420a8001df80ecfa937f10c2577488c4db7e2f626b9f60560dec0fc/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/9bf41862bb0f64ef736e62b836d2061395b97fc7861bb21e1fdebe0f213e927f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/1736ace7b9618a8f7821517f83489603a8f4709d0d24b7efd3b69bc7a2435140/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/8d75cce7e8c0055b5689fc06967b493fe6830482436adc24e629993f23f45801/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/5439908d1fd73645731c0854ed2991327fbc9638894a40395b0ed2f09166d125/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/0432c08492f13fa1465c5674bafa9a068ac3fdb7b963b97f6da8a7344dab8b7e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e44a052c6cd39f826c11d6d7a5ef4019f01ebf59e2c3877ccec7e51b4510debf/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/28e166ad7bddc9706eb8d0bbc5d98e54e69012c5da417477fc07efbfc54a5021/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/5565de5c5d761d0e99097a7dede880952906febcea1ff3fb151990ede17668f3/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e084932ef6ed798c3f29e1747dc5ff50385962816e37d64ba5e089fa9f95b55e/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/29c2aae8bb7b2f8f7edf5c7da6289207b9c9881f5447fb8de1cb8df62af625d5/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/e4729886c1888212775f0f27c660ede2bdea6181e6e619210ef4e89509e068d6/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/785627ffb460df6e42741f91209d9249d01cbe3282e5de04c15f44d75d56533f/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/dc2f163b14a2ced1a6de9e57a3330fb5dd10e33de8c0efe6ecf234e15e18175c/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/37f3942c95496cda9eec224a77eed9fcc9cf23ff242d59e13e56c5fd12e17c8b/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c555c0f82bc7e3d552862f0f4cf6b94d483839a1e481a6a04783e4fd71cb8911/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/39ac7372a1531ed28ca675c99965b791f7debf6f8d23e875f765bd0fc433cfb6/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/bdbcbd637337b832e59ccefda91083c2df8d41c45d1d80b945de929ea24d42a4/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d07f796e8bf8c6e53dac2c44d8c1cc135146c9d8014ef3bfd6cfd39a74e6e7a7/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/5c04cb52bfa3ad1e38643233a6c061f72c8910945f8618ba878543c3e79508c9/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/be67816c77b9477efbdccd8b24f0452290059d53889b6fe93eb01c22e6d49e19/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/c3114990c48f88c3ceaa5d676b96aaea64211fe1bdd1e8ef83b88b14f82ce07e/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/b2091d1381e4820e62f90add315068e7cbf67e2e93b74ce50b503e6f4150036e/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/5e9acf873677b71496f2f96dff58ce907fe652ee5abacdf809f3d432b5d35858/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/7b3a2e87a49cb571d94b3416fbb3ef521d25aecec28454fdcd47d86d06e31a1a/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/979fef97c88be9f698e09f2d1694d2ec9cffbfe2c31112efb523f4d31763337f/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/ef7ddd8083ac08f63a5bcf8e0f40178befad9f42e062ed701dcbe56fe5f4b7dc/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/e9a6915022db201ab860beb4765bf1071bc4430e85ddccac7016eed21927644a/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f0926fa6df5956321b89ce450d8face32205a48d27bfa49b573f8236d15c0634/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/bdabb68e7be89eaa14dfffa6069284490d344c6204a7852f56464c3c0afe3c92/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/42366c20c21ccc7075886b2ffd73fe576da0945ba9ebebdc702c3359f78beda6/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/16dcd2dfa7373c3e75c4e94f178eee8817de37946265693bdd17d60c88e5086f/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/224003c7adfbadee8a26a338351b239a6cdb4f78ab538668008e6d3fc4cdf7b9/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/f748b8b0a780d912763722dcfa6d382b92a6aa8bcc434b81593b4ee60d8ba106/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/54b99a23ee5d1759130a84853b56a3c66840f795d75956eb58e1991e105b218f/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8cfddc32918aacda83a5fe0d84dfcaa9264a8b47d33d8abe449004d119d58bb7/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-71:{mountpoint:/var/lib/containers/storage/overlay/1e8e5bf1330975ef3ea9659c627078bb3b6b174ad32223f0986168fb66b4f81c/merged major:0 minor:71 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/746062df2179a3fa26d7101d2242af9e4cec66ff972ab31740df9290162b9bca/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f58d2574667d73f147e2f01ffba8edfe3b94ef5cbc1f59fcd91622ff9b395b63/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/0b096d1db6d4c90ae17d5c13ecffa771c22905886948aed7a3e9b63742ee699f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/245f2086be9840680495f6dd827e2b72b1ea6ba3a7565c4495bf328b0061fbff/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/466af8fea7fd20150e0bec707134e469058bb7c9978ecca29f9bd91548fc2844/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4515463b872da302bf075d44f8e867ee27fe098d2cc30d8fd751932af291ffe2/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9e37dc7410fb229986fc58cf451f94c01ebaf765bb05d9d174a4133e1612a3b7/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/dab160846ea395bb8d2dc423ab5f0027ac4af7254d0c5014f7dd884ab274b344/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 31 07:29:15.609277 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.608662 4870 manager.go:217] Machine: {Timestamp:2026-10-31 07:29:15.607485584 +0000 UTC m=+0.079879824 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099833 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c69ced50f30df5156ec7e67561d9f SystemUUID:ec2c69ce-d50f-30df-5156-ec7e67561d9f BootID:3a9a3864-1341-4820-b8e4-fbdea582fc0a Filesystems:[{Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~projected/kube-api-access-d9jxl DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ed8071017ab2adc6c2e3d9f60763fad5b160ee22a146353bc54cd9aa64e0b35/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~projected/kube-api-access-pwwmx DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5abb2415-7417-4478-9d12-a4d9f2e0549c/volumes/kubernetes.io~projected/kube-api-access-jndmz DeviceMajor:0 DeviceMinor:215 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4/volumes/kubernetes.io~projected/kube-api-access-82xjk DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3042b2eed940b569527df6683a1202ecc401ca58d1f567f1fbef64b71beadf85/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f52e908-ef31-4672-ac4d-1ee9778d1b52/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/20fefbf1-a4ae-4208-a927-2dc47132a88c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:210 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/148b73f8a99f2ee5f5c07a5219fe165d252d33cfcb6d1b932972de3dda5e7e21/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40edbaa014589fce59834e91bd341d8e8394bc18042d75d28bdbc4630c847615/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/8f52e908-ef31-4672-ac4d-1ee9778d1b52/volumes/kubernetes.io~projected/kube-api-access-9wjv9 DeviceMajor:0 DeviceMinor:224 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a122982a-c398-47f1-a368-fd89db8899a7/volumes/kubernetes.io~projected/kube-api-access-l6qvs DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89b66f1d54d1adaf3f3e1f465568263b5f01e7e03f9fd01a06ed500d2bea626b/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:219 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-71 DeviceMajor:0 DeviceMinor:71 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86e696ba16ccc6fcf6750fba79bdbf8374e1cfbb7a6a6d78a43a87fd54ee6d0e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d89c017bef7620d48a8c727dcc313e186bf52f4283e543b23a6f486d9755a4a/userdata/shm DeviceMajor:0 DeviceMinor:232 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6e9142e90d8f27e0a3b4126104b7b093a307aebb58b14205c2dacceb10e8b7e/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e/volumes/kubernetes.io~projected/kube-api-access-hs699 DeviceMajor:0 DeviceMinor:218 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b67627fb-3093-4fff-9ab1-c0a4432f72e2/volumes/kubernetes.io~projected/kube-api-access-lz678 DeviceMajor:0 DeviceMinor:228 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/078fca68-9d2f-4df5-924c-66d9c4198984/volumes/kubernetes.io~projected/kube-api-access-p94ln DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/805181c8-c83e-433d-9980-62f629c63f0d/volumes/kubernetes.io~projected/kube-api-access-7q5j8 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f71aef6466f43d853c31f06f4203de87fd03799f113513ea2a5e999a7f461b8/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c2ea6339b5c60358bae23ef1e9b76246a6d501c6c92e95d617b7e06bb6a5857/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~projected/kube-api-access-nkcmc DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2c31ebf7-17d4-4aef-824b-afd78bd52d6e/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:288 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volumes/kubernetes.io~projected/kube-api-access-8rklq DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:208 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:217 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e71bf711-37d2-421a-8bb5-a838f60d36dc/volumes/kubernetes.io~projected/kube-api-access-4r2jl DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0e2c4370-aeba-4e68-ad8d-396664e91a83/volumes/kubernetes.io~projected/kube-api-access-dsxxh DeviceMajor:0 DeviceMinor:318 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26ec4fb590749e8484eea542aa406a83abceb1143d9480374023800b7129b461/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6b34ebf-4617-4662-ad56-5dbbb97a4e68/volumes/kubernetes.io~projected/kube-api-access-qnr8n DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b163e33d-f1b0-4583-aeee-1e759841ef01/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/40c29f18ccc8b41e6341fc55d564b2ddac17895c21724b3d752944daf82b7e6c/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/92342503-edc9-45e9-b9d7-fff71ce5092d/volumes/kubernetes.io~projected/kube-api-access-9s7qq DeviceMajor:0 DeviceMinor:214 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4a619c0-1d45-4bfe-b948-f79f3d461cb2/volumes/kubernetes.io~projected/kube-api-access-l7txw DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9dacaea-1664-495c-b164-a939a91bac87/volumes/kubernetes.io~projected/kube-api-access-hb5h7 DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fca690a447d1365526a5a7e8fc2786d9e7bf645cb404deaad9726cbe136ba234/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4672f0309255cec153aa0970177f1dadde4b41e498604fddaee5a33136f7db4/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20fefbf1-a4ae-4208-a927-2dc47132a88c/volumes/kubernetes.io~projected/kube-api-access-vvjdz DeviceMajor:0 DeviceMinor:216 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/787267e5d40be1c3f543d534826e73743bb86107db30de9cc8f13ec4a3f2f589/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9e12a075-b4ec-44e3-8811-7b19f1410513/volumes/kubernetes.io~projected/kube-api-access-8jkrd DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/805181c8-c83e-433d-9980-62f629c63f0d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/93ba798efdab585b5bfc82cd6acec021368c3f7e1421621a2bc7ac8c8b499021/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/a52ee83a-3e9c-49e9-b599-ec9a43d40bde/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/306bb57eb3ae4766b2515fc3efe62bdccc18cffe818844e2c53028c98d654a48/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/af42539c-c267-4894-a291-764cbb70c30f/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/96548fcd-4945-49fd-a9d4-1577c47e0ed1/volumes/kubernetes.io~projected/kube-api-access-g8csv DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a36d3876-b5ff-48cd-913c-48108ed86f77/volumes/kubernetes.io~projected/kube-api-access-brsk7 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/874dd1db29f8549cb77797751d3fe2eaeb7d2be90ab18b26463bd2cf2bb1dd71/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4a619c0-1d45-4bfe-b948-f79f3d461cb2/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29d86a3e-2f5d-433b-835f-56df35aa18c2/volumes/kubernetes.io~projected/kube-api-access-8rqdj DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/35dff775-deae-46eb-a3bc-b5618173818f/volumes/kubernetes.io~projected/kube-api-access-vbfzg DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e71bf711-37d2-421a-8bb5-a838f60d36dc/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:205 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes/kubernetes.io~projected/kube-api-access-92hm9 DeviceMajor:0 DeviceMinor:221 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b096b8728ac848be1960d4960a2e9cd445d25850449ba092dd159dee583769c4/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5/volumes/kubernetes.io~projected/kube-api-access-rqcq5 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dfcad49145ec5f9a36f0fab77307980502b5cc4e0e64c1021d7ca72f75a4e78/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8039f97e-ddca-4960-80a9-b53c24950ca9/volumes/kubernetes.io~projected/kube-api-access-68gsc DeviceMajor:0 DeviceMinor:227 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0d89c017bef7620 MacAddress:02:e6:dd:91:47:29 Speed:10000 Mtu:8901} {Name:148b73f8a99f2ee MacAddress:fa:cf:0d:f5:0d:60 Speed:10000 Mtu:8901} {Name:26ec4fb590749e8 MacAddress:fa:ae:92:c8:8a:9b Speed:10000 Mtu:8901} {Name:5f71aef6466f43d MacAddress:fe:69:6c:a6:4f:ef Speed:10000 Mtu:8901} {Name:6ed8071017ab2ad MacAddress:12:c1:11:eb:da:d7 Speed:10000 Mtu:8901} {Name:787267e5d40be1c MacAddress:9e:e9:5b:42:c1:3f Speed:10000 Mtu:8901} {Name:89b66f1d54d1ada MacAddress:ae:c4:44:4e:fc:91 Speed:10000 Mtu:8901} {Name:93ba798efdab585 MacAddress:56:f8:0e:92:0e:3f Speed:10000 Mtu:8901} {Name:b096b8728ac848b MacAddress:aa:18:61:d5:81:f5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e7:10:32:1f Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:7e:e2:6e:73:ec Speed:0 Mtu:8901} {Name:e4672f0309255ce MacAddress:de:e4:49:54:6c:0d Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e7:10:32:1f Speed:0 Mtu:9001} {Name:f6e9142e90d8f27 MacAddress:ca:5f:8c:c8:19:4e Speed:10000 Mtu:8901} {Name:fca690a447d1365 MacAddress:fa:71:6b:08:99:9d Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:1e:9f:f8:f8:78:a2 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:2e:2c:3f:c4:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 07:29:15.609445 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609284 4870 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 07:29:15.609445 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609340 4870 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 07:29:15.609634 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609611 4870 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 07:29:15.609757 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609635 4870 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-6.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 07:29:15.609888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609759 4870 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 07:29:15.609888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609767 4870 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 07:29:15.609888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609777 4870 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 07:29:15.609888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609804 4870 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 07:29:15.609888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609836 4870 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 07:29:15.610039 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.609994 4870 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 07:29:15.610136 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.610127 4870 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 07:29:15.610164 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.610161 4870 kubelet.go:493] "Attempting to sync node with API server" Oct 31 07:29:15.610191 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.610171 4870 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 07:29:15.610191 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.610184 4870 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 07:29:15.610246 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.610193 4870 kubelet.go:405] "Adding apiserver pod source" Oct 31 07:29:15.610246 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.610204 4870 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 07:29:15.611097 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.611081 4870 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 07:29:15.611134 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.611099 4870 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 07:29:15.611616 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.611597 4870 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 07:29:15.613685 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.613671 4870 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 07:29:15.613685 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.613685 4870 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 07:29:15.613801 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.613782 4870 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 07:29:15.613988 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.613977 4870 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 07:29:15.614022 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614006 4870 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 07:29:15.614102 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614093 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 07:29:15.614132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614108 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 07:29:15.614132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614113 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 07:29:15.614132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614119 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 07:29:15.614132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614124 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614135 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614141 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614147 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614154 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614159 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614171 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614181 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614203 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 07:29:15.614234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.614209 4870 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 07:29:15.617493 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.617481 4870 server.go:1263] "Started kubelet" Oct 31 07:29:15.618067 ip-10-0-140-6 systemd[1]: Started Kubernetes Kubelet. Oct 31 07:29:15.618202 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.617554 4870 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 07:29:15.618335 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.617589 4870 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 07:29:15.618335 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.618247 4870 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 07:29:15.618572 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.618538 4870 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 07:29:15.619246 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.619230 4870 server.go:317] "Adding debug handlers to kubelet server" Oct 31 07:29:15.622297 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.622279 4870 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 07:29:15.622785 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.622615 4870 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 07:29:15.622785 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:15.622648 4870 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-6.ec2.internal\" not found" Oct 31 07:29:15.622991 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.622852 4870 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 07:29:15.622991 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.622877 4870 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 07:29:15.623136 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.622619 4870 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 07:29:15.623136 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.623085 4870 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 07:23:37 +0000 UTC" deadline="2028-04-20 20:51:31.554781874 +0000 UTC" Oct 31 07:29:15.623136 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.623122 4870 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12901h22m15.931663914s" Oct 31 07:29:15.624312 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.624219 4870 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 07:29:15.624312 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.624238 4870 factory.go:55] Registering systemd factory Oct 31 07:29:15.624312 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.624247 4870 factory.go:223] Registration of the systemd container factory successfully Oct 31 07:29:15.625953 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.625936 4870 factory.go:153] Registering CRI-O factory Oct 31 07:29:15.625953 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.625955 4870 factory.go:223] Registration of the crio container factory successfully Oct 31 07:29:15.626088 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.625973 4870 factory.go:103] Registering Raw factory Oct 31 07:29:15.626088 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.625987 4870 manager.go:1199] Started watching for new ooms in manager Oct 31 07:29:15.626295 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626282 4870 manager.go:319] Starting recovery of all containers Oct 31 07:29:15.626581 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626546 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl" seLinuxMountContext="" Oct 31 07:29:15.626648 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626588 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9" seLinuxMountContext="" Oct 31 07:29:15.626648 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626604 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8039f97e-ddca-4960-80a9-b53c24950ca9" volumeName="kubernetes.io/configmap/8039f97e-ddca-4960-80a9-b53c24950ca9-config-volume" seLinuxMountContext="" Oct 31 07:29:15.626648 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626620 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e12a075-b4ec-44e3-8811-7b19f1410513" volumeName="kubernetes.io/projected/9e12a075-b4ec-44e3-8811-7b19f1410513-kube-api-access-8jkrd" seLinuxMountContext="" Oct 31 07:29:15.626648 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626635 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af42539c-c267-4894-a291-764cbb70c30f" volumeName="kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-default-certificate" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626649 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb98ee1-3f52-4241-b8a9-c76a19ade7a5" volumeName="kubernetes.io/configmap/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-iptables-alerter-script" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626662 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20fefbf1-a4ae-4208-a927-2dc47132a88c" volumeName="kubernetes.io/configmap/20fefbf1-a4ae-4208-a927-2dc47132a88c-config" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626675 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626691 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f52e908-ef31-4672-ac4d-1ee9778d1b52" volumeName="kubernetes.io/projected/8f52e908-ef31-4672-ac4d-1ee9778d1b52-kube-api-access-9wjv9" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626704 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b163e33d-f1b0-4583-aeee-1e759841ef01" volumeName="kubernetes.io/secret/b163e33d-f1b0-4583-aeee-1e759841ef01-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626716 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb684428-62fd-4dc0-aa16-3ba4dfc7c89b" volumeName="kubernetes.io/secret/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-agent-certs" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626730 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a122982a-c398-47f1-a368-fd89db8899a7" volumeName="kubernetes.io/projected/a122982a-c398-47f1-a368-fd89db8899a7-kube-api-access-l6qvs" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626743 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e2c4370-aeba-4e68-ad8d-396664e91a83" volumeName="kubernetes.io/projected/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-api-access-dsxxh" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626755 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8039f97e-ddca-4960-80a9-b53c24950ca9" volumeName="kubernetes.io/projected/8039f97e-ddca-4960-80a9-b53c24950ca9-kube-api-access-68gsc" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626770 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f52e908-ef31-4672-ac4d-1ee9778d1b52" volumeName="kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-service-ca-bundle" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626784 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f52e908-ef31-4672-ac4d-1ee9778d1b52" volumeName="kubernetes.io/secret/8f52e908-ef31-4672-ac4d-1ee9778d1b52-serving-cert" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626813 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" volumeName="kubernetes.io/secret/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-serving-cert" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626827 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626839 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626852 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805181c8-c83e-433d-9980-62f629c63f0d" volumeName="kubernetes.io/configmap/805181c8-c83e-433d-9980-62f629c63f0d-config" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626865 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f52e908-ef31-4672-ac4d-1ee9778d1b52" volumeName="kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-trusted-ca-bundle" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626878 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b163e33d-f1b0-4583-aeee-1e759841ef01" volumeName="kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-config" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626890 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29d86a3e-2f5d-433b-835f-56df35aa18c2" volumeName="kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-cni-binary-copy" seLinuxMountContext="" Oct 31 07:29:15.626902 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626904 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29d86a3e-2f5d-433b-835f-56df35aa18c2" volumeName="kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-daemon-config" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626916 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9dacaea-1664-495c-b164-a939a91bac87" volumeName="kubernetes.io/projected/b9dacaea-1664-495c-b164-a939a91bac87-kube-api-access-hb5h7" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626928 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626942 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f52e908-ef31-4672-ac4d-1ee9778d1b52" volumeName="kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-snapshots" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626954 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a36d3876-b5ff-48cd-913c-48108ed86f77" volumeName="kubernetes.io/projected/a36d3876-b5ff-48cd-913c-48108ed86f77-kube-api-access-brsk7" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626970 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a52ee83a-3e9c-49e9-b599-ec9a43d40bde" volumeName="kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-tuned" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626984 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="078fca68-9d2f-4df5-924c-66d9c4198984" volumeName="kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.626996 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e2c4370-aeba-4e68-ad8d-396664e91a83" volumeName="kubernetes.io/empty-dir/0e2c4370-aeba-4e68-ad8d-396664e91a83-data-volume" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627010 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29d86a3e-2f5d-433b-835f-56df35aa18c2" volumeName="kubernetes.io/projected/29d86a3e-2f5d-433b-835f-56df35aa18c2-kube-api-access-8rqdj" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627022 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8039f97e-ddca-4960-80a9-b53c24950ca9" volumeName="kubernetes.io/empty-dir/8039f97e-ddca-4960-80a9-b53c24950ca9-tmp-dir" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627035 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af42539c-c267-4894-a291-764cbb70c30f" volumeName="kubernetes.io/projected/af42539c-c267-4894-a291-764cbb70c30f-kube-api-access-nkcmc" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627047 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb684428-62fd-4dc0-aa16-3ba4dfc7c89b" volumeName="kubernetes.io/configmap/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-konnectivity-ca" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627061 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e" volumeName="kubernetes.io/projected/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e-kube-api-access-hs699" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627076 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627088 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92342503-edc9-45e9-b9d7-fff71ce5092d" volumeName="kubernetes.io/configmap/92342503-edc9-45e9-b9d7-fff71ce5092d-telemetry-config" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627100 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cb98ee1-3f52-4241-b8a9-c76a19ade7a5" volumeName="kubernetes.io/projected/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-kube-api-access-rqcq5" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627113 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d8cbe30-d117-4aa4-92ed-e7e81544a1f4" volumeName="kubernetes.io/projected/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kube-api-access-82xjk" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627124 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" volumeName="kubernetes.io/projected/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-kube-api-access-l7txw" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627138 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a52ee83a-3e9c-49e9-b599-ec9a43d40bde" volumeName="kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-tmp" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627150 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b163e33d-f1b0-4583-aeee-1e759841ef01" volumeName="kubernetes.io/projected/b163e33d-f1b0-4583-aeee-1e759841ef01-kube-api-access-8rklq" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627162 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="078fca68-9d2f-4df5-924c-66d9c4198984" volumeName="kubernetes.io/projected/078fca68-9d2f-4df5-924c-66d9c4198984-kube-api-access-p94ln" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627173 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20fefbf1-a4ae-4208-a927-2dc47132a88c" volumeName="kubernetes.io/secret/20fefbf1-a4ae-4208-a927-2dc47132a88c-serving-cert" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627205 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35dff775-deae-46eb-a3bc-b5618173818f" volumeName="kubernetes.io/projected/35dff775-deae-46eb-a3bc-b5618173818f-kube-api-access-vbfzg" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627217 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fec1a19-aa94-423d-a573-71c8b0739f55" volumeName="kubernetes.io/configmap/5fec1a19-aa94-423d-a573-71c8b0739f55-nginx-conf" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627229 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805181c8-c83e-433d-9980-62f629c63f0d" volumeName="kubernetes.io/projected/805181c8-c83e-433d-9980-62f629c63f0d-kube-api-access-7q5j8" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627241 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="805181c8-c83e-433d-9980-62f629c63f0d" volumeName="kubernetes.io/secret/805181c8-c83e-433d-9980-62f629c63f0d-serving-cert" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627253 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6b34ebf-4617-4662-ad56-5dbbb97a4e68" volumeName="kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627263 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" volumeName="kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-trusted-ca" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627274 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20fefbf1-a4ae-4208-a927-2dc47132a88c" volumeName="kubernetes.io/projected/20fefbf1-a4ae-4208-a927-2dc47132a88c-kube-api-access-vvjdz" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627285 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627297 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f52e908-ef31-4672-ac4d-1ee9778d1b52" volumeName="kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-tmp" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627308 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92342503-edc9-45e9-b9d7-fff71ce5092d" volumeName="kubernetes.io/projected/92342503-edc9-45e9-b9d7-fff71ce5092d-kube-api-access-9s7qq" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627319 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af42539c-c267-4894-a291-764cbb70c30f" volumeName="kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-stats-auth" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627337 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="078fca68-9d2f-4df5-924c-66d9c4198984" volumeName="kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627348 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96548fcd-4945-49fd-a9d4-1577c47e0ed1" volumeName="kubernetes.io/projected/96548fcd-4945-49fd-a9d4-1577c47e0ed1-kube-api-access-g8csv" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627361 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e12a075-b4ec-44e3-8811-7b19f1410513" volumeName="kubernetes.io/configmap/9e12a075-b4ec-44e3-8811-7b19f1410513-serviceca" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627373 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b163e33d-f1b0-4583-aeee-1e759841ef01" volumeName="kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-script-lib" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627384 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b67627fb-3093-4fff-9ab1-c0a4432f72e2" volumeName="kubernetes.io/projected/b67627fb-3093-4fff-9ab1-c0a4432f72e2-kube-api-access-lz678" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627396 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="078fca68-9d2f-4df5-924c-66d9c4198984" volumeName="kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-binary-copy" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627407 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627418 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627430 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627442 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="245742f2-7508-4558-93cf-42487bedc917" volumeName="kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627452 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c31ebf7-17d4-4aef-824b-afd78bd52d6e" volumeName="kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627464 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5abb2415-7417-4478-9d12-a4d9f2e0549c" volumeName="kubernetes.io/projected/5abb2415-7417-4478-9d12-a4d9f2e0549c-kube-api-access-jndmz" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627476 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" volumeName="kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-config" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627486 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a52ee83a-3e9c-49e9-b599-ec9a43d40bde" volumeName="kubernetes.io/projected/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-kube-api-access-pwwmx" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627497 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b163e33d-f1b0-4583-aeee-1e759841ef01" volumeName="kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-env-overrides" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627509 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627520 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ae4c917-214d-43eb-b14e-f58c5a8e755f" volumeName="kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627532 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="96548fcd-4945-49fd-a9d4-1577c47e0ed1" volumeName="kubernetes.io/configmap/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-config" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627545 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a122982a-c398-47f1-a368-fd89db8899a7" volumeName="kubernetes.io/empty-dir/a122982a-c398-47f1-a368-fd89db8899a7-tmp-dir" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627572 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e71bf711-37d2-421a-8bb5-a838f60d36dc" volumeName="kubernetes.io/projected/e71bf711-37d2-421a-8bb5-a838f60d36dc-kube-api-access-4r2jl" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627584 4870 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e71bf711-37d2-421a-8bb5-a838f60d36dc" volumeName="kubernetes.io/secret/e71bf711-37d2-421a-8bb5-a838f60d36dc-cert" seLinuxMountContext="" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627597 4870 reconstruct.go:97] "Volume reconstruction finished" Oct 31 07:29:15.627766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.627604 4870 reconciler.go:29] "Reconciler: start to sync state" Oct 31 07:29:15.641381 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.641282 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/1.log" Oct 31 07:29:15.642075 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.642044 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/1.log" Oct 31 07:29:15.654127 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.654080 4870 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 07:29:15.655610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.655585 4870 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 07:29:15.655610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.655610 4870 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 07:29:15.655761 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.655632 4870 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 07:29:15.655761 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:15.655678 4870 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 07:29:15.673398 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.673374 4870 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:29:15.674126 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.673979 4870 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:29:15.674126 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.674101 4870 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 07:29:15.680737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.680713 4870 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="4fe3aa1422601672dcaa52ee2d0365c94885958a5482699f9a2ca4c1df83bc9d" exitCode=0 Oct 31 07:29:15.680737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.680735 4870 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="3cb3c765f6668039d7e6fc85b990d843569a357319f52ed71eb03b3914e5e5e1" exitCode=0 Oct 31 07:29:15.680869 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.680743 4870 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="da07b725a9e32d6f1cd6b68c0e94861efb60fc5d6da554c1cf6fd080e18c8d20" exitCode=0 Oct 31 07:29:15.680869 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.680749 4870 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="5bf27ca0c80a224c76b95940b677516114f5e52f37cd45be8ff303bed7ec0e23" exitCode=0 Oct 31 07:29:15.680869 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.680755 4870 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="67a826f450028581be5d153374894f56b01f7e53dac3c73b76ce8fbbd2dad779" exitCode=0 Oct 31 07:29:15.680869 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.680760 4870 generic.go:356] "Generic (PLEG): container finished" podID="078fca68-9d2f-4df5-924c-66d9c4198984" containerID="711ae334f61129f3d8eab456920d7620c9c1ed35d3b23956624581c22fca4340" exitCode=0 Oct 31 07:29:15.685067 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.685048 4870 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:29:15.685275 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.685257 4870 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 07:29:15.688913 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.688898 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/1.log" Oct 31 07:29:15.689001 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.688918 4870 generic.go:356] "Generic (PLEG): container finished" podID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" containerID="0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4" exitCode=255 Oct 31 07:29:15.705814 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.705767 4870 generic.go:356] "Generic (PLEG): container finished" podID="3208be082c57abb809b875a8b99d7ba5" containerID="5bb838a3df977cd37a557a0563db09cbc7dfa34f5359aeefff920eb05f34e779" exitCode=0 Oct 31 07:29:15.716145 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.716128 4870 manager.go:324] Recovery completed Oct 31 07:29:15.735209 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735162 4870 cpu_manager.go:226] "Starting" policy="none" Oct 31 07:29:15.735209 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735177 4870 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 07:29:15.735209 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735197 4870 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 07:29:15.735358 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735342 4870 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 07:29:15.735396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735361 4870 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 07:29:15.735396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735383 4870 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 07:29:15.735396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735391 4870 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 07:29:15.735396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735398 4870 policy_none.go:50] "Start" Oct 31 07:29:15.735497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735407 4870 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 07:29:15.735497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735417 4870 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 07:29:15.735552 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735544 4870 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 07:29:15.735552 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735551 4870 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 07:29:15.735607 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.735557 4870 policy_none.go:44] "Start" Oct 31 07:29:15.739855 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.739830 4870 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 07:29:15.742410 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.742396 4870 manager.go:341] "Starting Device Plugin manager" Oct 31 07:29:15.742874 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:15.742421 4870 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 07:29:15.742928 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.742876 4870 server.go:87] "Starting device plugin registration server" Oct 31 07:29:15.743109 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.743100 4870 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 07:29:15.743143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.743110 4870 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 07:29:15.743228 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.743210 4870 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 07:29:15.743929 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.743890 4870 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 07:29:15.744021 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.743938 4870 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 07:29:15.744021 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.743946 4870 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 07:29:15.744115 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.744016 4870 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T07:29:15.743926878Z","UUID":"d5ba7497-e459-4a9d-977c-48fc6e892b82","Handler":null,"Name":"","Endpoint":""} Oct 31 07:29:15.745769 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:15.745731 4870 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 07:29:15.745769 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.745748 4870 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 07:29:15.745975 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.745777 4870 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 07:29:15.755809 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.755756 4870 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal"] Oct 31 07:29:15.756461 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756422 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"3208be082c57abb809b875a8b99d7ba5","Type":"ContainerStarted","Data":"defc31f8355364286d8f2480338e060715a8252d5cf93daa01d87eecd3989a6d"} Oct 31 07:29:15.756561 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756475 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"3208be082c57abb809b875a8b99d7ba5","Type":"ContainerDied","Data":"5bb838a3df977cd37a557a0563db09cbc7dfa34f5359aeefff920eb05f34e779"} Oct 31 07:29:15.756561 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756495 4870 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.756561 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756535 4870 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.756708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756500 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" event={"ID":"3208be082c57abb809b875a8b99d7ba5","Type":"ContainerStarted","Data":"06e4d196863fabaffbbfcd780de514c65a45336c8ad02364560a7a0036bcd305"} Oct 31 07:29:15.756708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756630 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" event={"ID":"8952dcaf440f7a0e696c251b7a42e952","Type":"ContainerStarted","Data":"79e52bf58fe37bb1c959dee4a387635ac916c00515ca8cc1f64dd18a2c779481"} Oct 31 07:29:15.756708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.756660 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" event={"ID":"8952dcaf440f7a0e696c251b7a42e952","Type":"ContainerStarted","Data":"9d1aa60f9c6c3bd540e853c7321f2495e056739ae6600eaea9a55cb3e088a4b6"} Oct 31 07:29:15.780727 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.780706 4870 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 07:29:15.780813 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:15.780749 4870 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.781336 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.781317 4870 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 07:29:15.781379 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:15.781367 4870 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.843916 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.843888 4870 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 07:29:15.845297 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.845275 4870 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientMemory" Oct 31 07:29:15.845437 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.845315 4870 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasNoDiskPressure" Oct 31 07:29:15.845437 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.845331 4870 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-6.ec2.internal" event="NodeHasSufficientPID" Oct 31 07:29:15.845437 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.845383 4870 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.875242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.875219 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.875242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.875248 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8952dcaf440f7a0e696c251b7a42e952-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"8952dcaf440f7a0e696c251b7a42e952\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.875370 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.875265 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.880007 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.879990 4870 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.880071 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.880049 4870 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.976317 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.976287 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.976472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.976322 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.976472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.976345 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8952dcaf440f7a0e696c251b7a42e952-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"8952dcaf440f7a0e696c251b7a42e952\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.976472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.976384 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.976472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.976400 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3208be082c57abb809b875a8b99d7ba5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" (UID: \"3208be082c57abb809b875a8b99d7ba5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:15.976472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:15.976393 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/8952dcaf440f7a0e696c251b7a42e952-config\") pod \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" (UID: \"8952dcaf440f7a0e696c251b7a42e952\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:16.610603 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.610560 4870 apiserver.go:52] "Watching apiserver" Oct 31 07:29:16.668371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.668334 4870 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 07:29:16.668970 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.668946 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m99x4","kube-system/konnectivity-agent-2qrrp","openshift-multus/network-metrics-daemon-qnzx5","openshift-multus/multus-additional-cni-plugins-btjmv","openshift-cluster-node-tuning-operator/tuned-wxjpf","kube-system/global-pull-secret-syncer-l5c85","kube-system/kas-connection-checker-685895c7c9-v5stm","openshift-image-registry/image-registry-546c4cd566-xfkx2","openshift-network-console/networking-console-plugin-86c478bb88-p9pnl","openshift-console-operator/console-operator-7cd56b8dbc-hzng6","openshift-ingress-canary/ingress-canary-g29bz","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj","openshift-insights/insights-runtime-extractor-zdg8b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz","openshift-insights/insights-operator-7c4986d449-vvm6c","openshift-multus/multus-qspqb","openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc","openshift-dns/node-resolver-sv7zk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal","openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5","openshift-ingress/router-default-5b4c78c6d5-pmxm4","kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal","openshift-image-registry/image-registry-57d7689f6c-h8wmf","kube-system/kas-connection-checker-685895c7c9-6sww9","kube-system/kas-connection-checker-685895c7c9-xzc9s","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc","openshift-image-registry/node-ca-m8msg","openshift-network-operator/iptables-alerter-wwhpb","openshift-ovn-kubernetes/ovnkube-node-2b2kf","openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk","openshift-network-diagnostics/network-check-target-gdgxp"] Oct 31 07:29:16.670153 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.670130 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.671038 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.671016 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.671381 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.671365 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.673031 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.673014 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.673436 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.673420 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:16.676982 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.676967 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:16.679366 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.679347 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.683016 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.682994 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.683720 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.683702 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.683720 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.683709 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.683888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.683818 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.684081 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.684062 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.684154 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.684076 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 31 07:29:16.684288 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.684261 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 31 07:29:16.694128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.694104 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.694128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.694103 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 07:29:16.694128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.694104 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 31 07:29:16.694307 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.694195 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-jhgwz\"" Oct 31 07:29:16.694307 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.694242 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 31 07:29:16.696487 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.696469 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-mwmjb\"" Oct 31 07:29:16.696816 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.696787 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 31 07:29:16.696879 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.696818 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 31 07:29:16.696879 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.696824 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 31 07:29:16.697136 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697120 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rhfwc\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697160 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697161 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-lzwlj\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697163 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-tdhmc\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697199 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697200 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9v74w\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697214 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 31 07:29:16.697232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697228 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697243 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697264 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-w856f\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697250 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697245 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697266 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697280 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697313 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697324 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-w4rqx\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697366 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-xcjxx\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697448 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697454 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697480 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697499 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-krpl2\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697564 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 31 07:29:16.697609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697596 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697624 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-ddkhs\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697644 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697678 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hr94j\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697730 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qvcsq\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697787 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dvd7d\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697859 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mrcbt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.697925 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-cp4x6\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698023 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698035 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698074 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698076 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-48xxx\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698094 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698079 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698107 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.698243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698130 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.699050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698909 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.699050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698921 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.699050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698932 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 07:29:16.699050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698925 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 07:29:16.699050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.698935 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 07:29:16.699050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.699047 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 31 07:29:16.699306 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.699094 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.699306 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.699125 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 07:29:16.699306 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.699230 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.699306 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.699277 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.699306 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.699287 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.707763 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707745 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.707875 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707844 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.707875 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707866 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707892 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707905 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707935 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707938 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707946 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707955 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707963 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707977 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 07:29:16.707987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707984 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.707999 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708003 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708016 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708030 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708031 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708048 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708056 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 31 07:29:16.708299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708075 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.708847 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708815 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 31 07:29:16.708847 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708829 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.708987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708851 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 07:29:16.708987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708895 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 07:29:16.708987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708904 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.708987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708953 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.708987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708980 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 07:29:16.709186 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708984 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.709186 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.708982 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.709409 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.709394 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 07:29:16.710190 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.710166 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 07:29:16.710668 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.710641 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 31 07:29:16.711255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.711039 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 07:29:16.711683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.711667 4870 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:16.711865 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.711837 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 31 07:29:16.712137 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.712121 4870 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:16.724856 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.724838 4870 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 07:29:16.734520 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.734505 4870 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 07:29:16.734597 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.734542 4870 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-6.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-6.ec2.internal" Oct 31 07:29:16.734990 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.734969 4870 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 07:29:16.735078 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.735010 4870 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-6.ec2.internal" Oct 31 07:29:16.778731 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778707 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-systemd-units\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.778845 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778743 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/805181c8-c83e-433d-9980-62f629c63f0d-config\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:16.778845 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778771 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.778962 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778846 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-konnectivity-ca\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:29:16.778962 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778873 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-iptables-alerter-script\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.778962 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778890 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-stats-auth\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.778962 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778910 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.778962 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778934 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.778962 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.778959 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a122982a-c398-47f1-a368-fd89db8899a7-hosts-file\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.779216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779059 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/805181c8-c83e-433d-9980-62f629c63f0d-config\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:16.779216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779089 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.779216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779120 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-konnectivity-ca\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:29:16.779216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779121 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-etc-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.779216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779198 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.779216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779211 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71bf711-37d2-421a-8bb5-a838f60d36dc-cert\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779211 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-iptables-alerter-script\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779260 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5j8\" (UniqueName: \"kubernetes.io/projected/805181c8-c83e-433d-9980-62f629c63f0d-kube-api-access-7q5j8\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779288 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-socket-dir-parent\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779293 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-trusted-ca\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779307 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs699\" (UniqueName: \"kubernetes.io/projected/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e-kube-api-access-hs699\") pod \"kas-connection-checker-685895c7c9-6sww9\" (UID: \"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e\") " pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779338 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-multus-certs\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779356 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-stats-auth\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779401 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-kubelet-config\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779469 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.779515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779471 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e71bf711-37d2-421a-8bb5-a838f60d36dc-cert\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779564 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f52e908-ef31-4672-ac4d-1ee9778d1b52-serving-cert\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779683 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-default-certificate\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779717 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-multus\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779728 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779739 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8f52e908-ef31-4672-ac4d-1ee9778d1b52-serving-cert\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779746 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779845 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8039f97e-ddca-4960-80a9-b53c24950ca9-config-volume\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779890 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e12a075-b4ec-44e3-8811-7b19f1410513-host\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.779939 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779924 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779952 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779950 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-default-certificate\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.779979 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-modprobe-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780008 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780010 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-lib-modules\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780045 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-hostroot\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780043 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8039f97e-ddca-4960-80a9-b53c24950ca9-config-volume\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780069 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780097 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82xjk\" (UniqueName: \"kubernetes.io/projected/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kube-api-access-82xjk\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780124 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a122982a-c398-47f1-a368-fd89db8899a7-tmp-dir\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780206 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780220 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-systemd\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780239 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a122982a-c398-47f1-a368-fd89db8899a7-tmp-dir\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780264 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-run\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780292 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780341 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-cnibin\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.780357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780366 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkrd\" (UniqueName: \"kubernetes.io/projected/9e12a075-b4ec-44e3-8811-7b19f1410513-kube-api-access-8jkrd\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780381 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-netd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780399 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780563 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780582 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780598 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-tuned\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780613 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-config\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780642 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brsk7\" (UniqueName: \"kubernetes.io/projected/a36d3876-b5ff-48cd-913c-48108ed86f77-kube-api-access-brsk7\") pod \"network-check-source-6f78d59f99-ll9q5\" (UID: \"a36d3876-b5ff-48cd-913c-48108ed86f77\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780668 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8039f97e-ddca-4960-80a9-b53c24950ca9-tmp-dir\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780686 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-tuned\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780691 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb5h7\" (UniqueName: \"kubernetes.io/projected/b9dacaea-1664-495c-b164-a939a91bac87-kube-api-access-hb5h7\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780724 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780752 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8csv\" (UniqueName: \"kubernetes.io/projected/96548fcd-4945-49fd-a9d4-1577c47e0ed1-kube-api-access-g8csv\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780766 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8039f97e-ddca-4960-80a9-b53c24950ca9-tmp-dir\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780769 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-original-pull-secret\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780777 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p94ln\" (UniqueName: \"kubernetes.io/projected/078fca68-9d2f-4df5-924c-66d9c4198984-kube-api-access-p94ln\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780820 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfzg\" (UniqueName: \"kubernetes.io/projected/35dff775-deae-46eb-a3bc-b5618173818f-kube-api-access-vbfzg\") pod \"kas-connection-checker-685895c7c9-xzc9s\" (UID: \"35dff775-deae-46eb-a3bc-b5618173818f\") " pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780845 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-system-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780886 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780941 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-config\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780940 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.780987 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781026 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-host\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781063 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwmx\" (UniqueName: \"kubernetes.io/projected/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-kube-api-access-pwwmx\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781094 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-env-overrides\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781119 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b163e33d-f1b0-4583-aeee-1e759841ef01-ovn-node-metrics-cert\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.781169 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781169 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781206 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781238 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqcq5\" (UniqueName: \"kubernetes.io/projected/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-kube-api-access-rqcq5\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781262 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781264 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781291 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-k8s-cni-cncf-io\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781345 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b163e33d-f1b0-4583-aeee-1e759841ef01-ovn-node-metrics-cert\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781708 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-env-overrides\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781777 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781961 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.781994 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gsc\" (UniqueName: \"kubernetes.io/projected/8039f97e-ddca-4960-80a9-b53c24950ca9-kube-api-access-68gsc\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782016 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782032 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782052 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-cnibin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.782118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782098 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782223 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysconfig\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782261 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6qvs\" (UniqueName: \"kubernetes.io/projected/a122982a-c398-47f1-a368-fd89db8899a7-kube-api-access-l6qvs\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782399 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782431 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-agent-certs\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782456 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782484 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782522 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-var-lib-kubelet\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782542 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-slash\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782559 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-var-lib-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782607 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-node-log\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782620 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-serving-cert\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782635 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-tmp\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782662 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782690 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782695 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782742 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-tmp\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782762 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eb684428-62fd-4dc0-aa16-3ba4dfc7c89b-agent-certs\") pod \"konnectivity-agent-2qrrp\" (UID: \"eb684428-62fd-4dc0-aa16-3ba4dfc7c89b\") " pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782828 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782847 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20fefbf1-a4ae-4208-a927-2dc47132a88c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782851 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782863 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/20fefbf1-a4ae-4208-a927-2dc47132a88c-kube-api-access-vvjdz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782884 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-cni-binary-copy\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782908 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782930 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.782976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782957 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-system-cni-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.782985 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-daemon-config\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783026 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-dbus\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783038 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/20fefbf1-a4ae-4208-a927-2dc47132a88c-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783060 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783085 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e12a075-b4ec-44e3-8811-7b19f1410513-serviceca\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783102 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-tmp\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783118 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e2c4370-aeba-4e68-ad8d-396664e91a83-crio-socket\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783135 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-binary-copy\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783141 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-cni-binary-copy\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783162 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783173 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-daemon-config\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783181 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-bin\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783196 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-netns\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783230 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-tmp\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783266 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-systemd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783299 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rklq\" (UniqueName: \"kubernetes.io/projected/b163e33d-f1b0-4583-aeee-1e759841ef01-kube-api-access-8rklq\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783337 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/805181c8-c83e-433d-9980-62f629c63f0d-serving-cert\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783353 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9e12a075-b4ec-44e3-8811-7b19f1410513-serviceca\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783443 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/078fca68-9d2f-4df5-924c-66d9c4198984-cni-binary-copy\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783501 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/805181c8-c83e-433d-9980-62f629c63f0d-serving-cert\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783506 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-snapshots\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783551 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/8f52e908-ef31-4672-ac4d-1ee9778d1b52-snapshots\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783556 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783583 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkcmc\" (UniqueName: \"kubernetes.io/projected/af42539c-c267-4894-a291-764cbb70c30f-kube-api-access-nkcmc\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.783596 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783608 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r2jl\" (UniqueName: \"kubernetes.io/projected/e71bf711-37d2-421a-8bb5-a838f60d36dc-kube-api-access-4r2jl\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783633 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-bin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783660 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783655 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-etc-kubernetes\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783721 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783742 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-service-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783778 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783813 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-kubelet\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783842 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783867 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-kubelet\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783892 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-netns\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783921 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7qq\" (UniqueName: \"kubernetes.io/projected/92342503-edc9-45e9-b9d7-fff71ce5092d-kube-api-access-9s7qq\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783943 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f52e908-ef31-4672-ac4d-1ee9778d1b52-service-ca-bundle\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.783950 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-conf-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784051 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsxxh\" (UniqueName: \"kubernetes.io/projected/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-api-access-dsxxh\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784084 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wjv9\" (UniqueName: \"kubernetes.io/projected/8f52e908-ef31-4672-ac4d-1ee9778d1b52-kube-api-access-9wjv9\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784118 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz678\" (UniqueName: \"kubernetes.io/projected/b67627fb-3093-4fff-9ab1-c0a4432f72e2-kube-api-access-lz678\") pod \"kas-connection-checker-685895c7c9-v5stm\" (UID: \"b67627fb-3093-4fff-9ab1-c0a4432f72e2\") " pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784190 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-conf\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784234 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784295 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e2c4370-aeba-4e68-ad8d-396664e91a83-data-volume\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784324 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-ovn\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784346 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-log-socket\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784403 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0e2c4370-aeba-4e68-ad8d-396664e91a83-data-volume\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784402 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rqdj\" (UniqueName: \"kubernetes.io/projected/29d86a3e-2f5d-433b-835f-56df35aa18c2-kube-api-access-8rqdj\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784434 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-kubernetes\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784452 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784469 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784485 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-config\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784500 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20fefbf1-a4ae-4208-a927-2dc47132a88c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784518 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784541 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-host-slash\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784585 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784627 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784637 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/20fefbf1-a4ae-4208-a927-2dc47132a88c-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784631 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784633 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-script-lib\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784661 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-config\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784671 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-device-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784701 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784724 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92342503-edc9-45e9-b9d7-fff71ce5092d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784757 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.784812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784838 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784859 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b163e33d-f1b0-4583-aeee-1e759841ef01-ovnkube-script-lib\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784863 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7txw\" (UniqueName: \"kubernetes.io/projected/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-kube-api-access-l7txw\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784926 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/92342503-edc9-45e9-b9d7-fff71ce5092d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.784996 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.785027 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-sys\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.785055 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fec1a19-aa94-423d-a573-71c8b0739f55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.785096 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndmz\" (UniqueName: \"kubernetes.io/projected/5abb2415-7417-4478-9d12-a4d9f2e0549c-kube-api-access-jndmz\") pod \"volume-data-source-validator-58c9f5f46d-cqqrp\" (UID: \"5abb2415-7417-4478-9d12-a4d9f2e0549c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:16.786391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.785257 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5fec1a19-aa94-423d-a573-71c8b0739f55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:16.802987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.802964 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs699\" (UniqueName: \"kubernetes.io/projected/33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e-kube-api-access-hs699\") pod \"kas-connection-checker-685895c7c9-6sww9\" (UID: \"33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e\") " pod="kube-system/kas-connection-checker-685895c7c9-6sww9" Oct 31 07:29:16.803068 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.802997 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5j8\" (UniqueName: \"kubernetes.io/projected/805181c8-c83e-433d-9980-62f629c63f0d-kube-api-access-7q5j8\") pod \"service-ca-operator-544d749c8c-lrjlk\" (UID: \"805181c8-c83e-433d-9980-62f629c63f0d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" Oct 31 07:29:16.803199 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.803108 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.803304 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.803286 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8csv\" (UniqueName: \"kubernetes.io/projected/96548fcd-4945-49fd-a9d4-1577c47e0ed1-kube-api-access-g8csv\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.803373 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.803349 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfzg\" (UniqueName: \"kubernetes.io/projected/35dff775-deae-46eb-a3bc-b5618173818f-kube-api-access-vbfzg\") pod \"kas-connection-checker-685895c7c9-xzc9s\" (UID: \"35dff775-deae-46eb-a3bc-b5618173818f\") " pod="kube-system/kas-connection-checker-685895c7c9-xzc9s" Oct 31 07:29:16.803619 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.803601 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkrd\" (UniqueName: \"kubernetes.io/projected/9e12a075-b4ec-44e3-8811-7b19f1410513-kube-api-access-8jkrd\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.803673 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.803635 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb5h7\" (UniqueName: \"kubernetes.io/projected/b9dacaea-1664-495c-b164-a939a91bac87-kube-api-access-hb5h7\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:16.804828 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.804810 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brsk7\" (UniqueName: \"kubernetes.io/projected/a36d3876-b5ff-48cd-913c-48108ed86f77-kube-api-access-brsk7\") pod \"network-check-source-6f78d59f99-ll9q5\" (UID: \"a36d3876-b5ff-48cd-913c-48108ed86f77\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-ll9q5" Oct 31 07:29:16.804931 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.804910 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.805077 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.805059 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.805125 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.805100 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6qvs\" (UniqueName: \"kubernetes.io/projected/a122982a-c398-47f1-a368-fd89db8899a7-kube-api-access-l6qvs\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.805165 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.805149 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqcq5\" (UniqueName: \"kubernetes.io/projected/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-kube-api-access-rqcq5\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.805204 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.805163 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p94ln\" (UniqueName: \"kubernetes.io/projected/078fca68-9d2f-4df5-924c-66d9c4198984-kube-api-access-p94ln\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.806074 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.806055 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr8n\" (UniqueName: \"kubernetes.io/projected/a6b34ebf-4617-4662-ad56-5dbbb97a4e68-kube-api-access-qnr8n\") pod \"network-check-target-gdgxp\" (UID: \"a6b34ebf-4617-4662-ad56-5dbbb97a4e68\") " pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:16.806394 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.806371 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwmx\" (UniqueName: \"kubernetes.io/projected/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-kube-api-access-pwwmx\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.808372 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.808355 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7qq\" (UniqueName: \"kubernetes.io/projected/92342503-edc9-45e9-b9d7-fff71ce5092d-kube-api-access-9s7qq\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.808933 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.808916 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkcmc\" (UniqueName: \"kubernetes.io/projected/af42539c-c267-4894-a291-764cbb70c30f-kube-api-access-nkcmc\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.809122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.809107 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r2jl\" (UniqueName: \"kubernetes.io/projected/e71bf711-37d2-421a-8bb5-a838f60d36dc-kube-api-access-4r2jl\") pod \"ingress-canary-g29bz\" (UID: \"e71bf711-37d2-421a-8bb5-a838f60d36dc\") " pod="openshift-ingress-canary/ingress-canary-g29bz" Oct 31 07:29:16.809837 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.809783 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvjdz\" (UniqueName: \"kubernetes.io/projected/20fefbf1-a4ae-4208-a927-2dc47132a88c-kube-api-access-vvjdz\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-njqtj\" (UID: \"20fefbf1-a4ae-4208-a927-2dc47132a88c\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" Oct 31 07:29:16.809948 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.809929 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsxxh\" (UniqueName: \"kubernetes.io/projected/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-api-access-dsxxh\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.809991 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.809932 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.810250 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.810234 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rqdj\" (UniqueName: \"kubernetes.io/projected/29d86a3e-2f5d-433b-835f-56df35aa18c2-kube-api-access-8rqdj\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.810584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.810570 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rklq\" (UniqueName: \"kubernetes.io/projected/b163e33d-f1b0-4583-aeee-1e759841ef01-kube-api-access-8rklq\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.812238 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.812219 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndmz\" (UniqueName: \"kubernetes.io/projected/5abb2415-7417-4478-9d12-a4d9f2e0549c-kube-api-access-jndmz\") pod \"volume-data-source-validator-58c9f5f46d-cqqrp\" (UID: \"5abb2415-7417-4478-9d12-a4d9f2e0549c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-cqqrp" Oct 31 07:29:16.812359 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.812342 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz678\" (UniqueName: \"kubernetes.io/projected/b67627fb-3093-4fff-9ab1-c0a4432f72e2-kube-api-access-lz678\") pod \"kas-connection-checker-685895c7c9-v5stm\" (UID: \"b67627fb-3093-4fff-9ab1-c0a4432f72e2\") " pod="kube-system/kas-connection-checker-685895c7c9-v5stm" Oct 31 07:29:16.812554 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.812537 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7txw\" (UniqueName: \"kubernetes.io/projected/a4a619c0-1d45-4bfe-b948-f79f3d461cb2-kube-api-access-l7txw\") pod \"console-operator-7cd56b8dbc-hzng6\" (UID: \"a4a619c0-1d45-4bfe-b948-f79f3d461cb2\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:16.812860 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.812844 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82xjk\" (UniqueName: \"kubernetes.io/projected/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kube-api-access-82xjk\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.814233 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.814217 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gsc\" (UniqueName: \"kubernetes.io/projected/8039f97e-ddca-4960-80a9-b53c24950ca9-kube-api-access-68gsc\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.817892 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.817865 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wjv9\" (UniqueName: \"kubernetes.io/projected/8f52e908-ef31-4672-ac4d-1ee9778d1b52-kube-api-access-9wjv9\") pod \"insights-operator-7c4986d449-vvm6c\" (UID: \"8f52e908-ef31-4672-ac4d-1ee9778d1b52\") " pod="openshift-insights/insights-operator-7c4986d449-vvm6c" Oct 31 07:29:16.885440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885385 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885416 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-var-lib-kubelet\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885439 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-slash\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885469 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885480 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-var-lib-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885493 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-var-lib-kubelet\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885507 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-node-log\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885527 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885547 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885564 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885570 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-node-log\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885545 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-slash\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.885588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885541 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-var-lib-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885579 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-system-cni-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885606 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-system-cni-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885619 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-dbus\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885624 4870 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885649 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e2c4370-aeba-4e68-ad8d-396664e91a83-crio-socket\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885657 4870 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885665 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-bin\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885671 4870 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885682 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-netns\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885705 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-systemd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885712 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-dbus\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885727 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-bin\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885728 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0e2c4370-aeba-4e68-ad8d-396664e91a83-crio-socket\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885659 4870 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885740 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-systemd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885740 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-netns\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885737 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.385717015 +0000 UTC m=+1.858111244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885842 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.385825508 +0000 UTC m=+1.858219738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.885860 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.385853908 +0000 UTC m=+1.858248133 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885872 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-bin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885888 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-etc-kubernetes\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885906 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885921 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-kubelet\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885937 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-kubelet\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885956 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-netns\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885960 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-etc-kubernetes\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885985 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-tuning-conf-dir\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885991 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-conf-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885998 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-bin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886009 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-kubelet\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886018 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-conf-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.885992 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-netns\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886020 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-conf\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886061 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-ovn\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886001 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-kubelet\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886082 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-log-socket\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886101 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-kubernetes\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886103 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-conf\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.886100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886114 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-run-ovn\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886130 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-log-socket\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886118 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886152 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886172 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-host-slash\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886194 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886209 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-device-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886241 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886248 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cb98ee1-3f52-4241-b8a9-c76a19ade7a5-host-slash\") pod \"iptables-alerter-wwhpb\" (UID: \"9cb98ee1-3f52-4241-b8a9-c76a19ade7a5\") " pod="openshift-network-operator/iptables-alerter-wwhpb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886242 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-device-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886264 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-kubernetes\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886273 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-sys-fs\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886272 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886294 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.386279458 +0000 UTC m=+1.858673695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886338 4870 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886398 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886433 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.386417441 +0000 UTC m=+1.858811671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886436 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886467 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886489 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-sys\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886505 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-systemd-units\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886521 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a122982a-c398-47f1-a368-fd89db8899a7-hosts-file\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886544 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-etc-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886570 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-sys\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886571 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-socket-dir-parent\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886583 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-registration-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886590 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-systemd-units\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886603 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-etc-openvswitch\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886604 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a122982a-c398-47f1-a368-fd89db8899a7-hosts-file\") pod \"node-resolver-sv7zk\" (UID: \"a122982a-c398-47f1-a368-fd89db8899a7\") " pod="openshift-dns/node-resolver-sv7zk" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886611 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-multus-certs\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886613 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-socket-dir-parent\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886639 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-kubelet-config\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886642 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-multus-certs\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886657 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-multus\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886672 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e12a075-b4ec-44e3-8811-7b19f1410513-host\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886699 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2c31ebf7-17d4-4aef-824b-afd78bd52d6e-kubelet-config\") pod \"global-pull-secret-syncer-l5c85\" (UID: \"2c31ebf7-17d4-4aef-824b-afd78bd52d6e\") " pod="kube-system/global-pull-secret-syncer-l5c85" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886700 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-var-lib-cni-multus\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886695 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886716 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9e12a075-b4ec-44e3-8811-7b19f1410513-host\") pod \"node-ca-m8msg\" (UID: \"9e12a075-b4ec-44e3-8811-7b19f1410513\") " pod="openshift-image-registry/node-ca-m8msg" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886736 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-modprobe-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886755 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-lib-modules\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886770 4870 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886804 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-hostroot\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886825 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886840 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.386827603 +0000 UTC m=+1.859221864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886853 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-modprobe-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886866 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-hostroot\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886875 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-lib-modules\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886868 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-systemd\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886886 4870 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886899 4870 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886909 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-systemd\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886923 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-run\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.886928 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.386919587 +0000 UTC m=+1.859313836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886898 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-run\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886975 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.886992 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-cnibin\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887010 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-netd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887028 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887042 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887061 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-system-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887099 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-cni-netd\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887120 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/078fca68-9d2f-4df5-924c-66d9c4198984-cnibin\") pod \"multus-additional-cni-plugins-btjmv\" (UID: \"078fca68-9d2f-4df5-924c-66d9c4198984\") " pod="openshift-multus/multus-additional-cni-plugins-btjmv" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.887132 4870 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887106 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887149 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-multus-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887160 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-host\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.887167 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.387155805 +0000 UTC m=+1.859550033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887196 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-system-cni-dir\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887201 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-host\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887205 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysctl-d\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887205 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887240 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887253 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-etc-selinux\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887269 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-k8s-cni-cncf-io\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887295 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887318 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887317 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-host-run-k8s-cni-cncf-io\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887333 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-cnibin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.887327 4870 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.887374 4870 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.887397 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.387382822 +0000 UTC m=+1.859777050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887374 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887395 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b163e33d-f1b0-4583-aeee-1e759841ef01-host-run-ovn-kubernetes\") pod \"ovnkube-node-2b2kf\" (UID: \"b163e33d-f1b0-4583-aeee-1e759841ef01\") " pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887378 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/29d86a3e-2f5d-433b-835f-56df35aa18c2-cnibin\") pod \"multus-qspqb\" (UID: \"29d86a3e-2f5d-433b-835f-56df35aa18c2\") " pod="openshift-multus/multus-qspqb" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:16.887424 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:17.38740636 +0000 UTC m=+1.859800589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : secret "metrics-daemon-secret" not found Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887437 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9d8cbe30-d117-4aa4-92ed-e7e81544a1f4-socket-dir\") pod \"aws-ebs-csi-driver-node-wmtvz\" (UID: \"9d8cbe30-d117-4aa4-92ed-e7e81544a1f4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wmtvz" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887460 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysconfig\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887529 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0e2c4370-aeba-4e68-ad8d-396664e91a83-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:16.887588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.887547 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a52ee83a-3e9c-49e9-b599-ec9a43d40bde-etc-sysconfig\") pod \"tuned-wxjpf\" (UID: \"a52ee83a-3e9c-49e9-b599-ec9a43d40bde\") " pod="openshift-cluster-node-tuning-operator/tuned-wxjpf" Oct 31 07:29:16.972961 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:16.972934 4870 scope.go:122] "RemoveContainer" containerID="0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4" Oct 31 07:29:17.392046 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.391947 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:17.392046 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.391998 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392059 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392100 4870 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392121 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392149 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392169 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392149449 +0000 UTC m=+2.864543679 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392214 4870 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392219 4870 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392238 4870 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392249 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392278 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392261992 +0000 UTC m=+2.864656236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:17.392286 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392227 4870 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392299 4870 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392302 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392292931 +0000 UTC m=+2.864687158 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392326 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls podName:92342503-edc9-45e9-b9d7-fff71ce5092d nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392316763 +0000 UTC m=+2.864710998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-vq7hc" (UID: "92342503-edc9-45e9-b9d7-fff71ce5092d") : secret "cluster-monitoring-operator-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392355 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392346843 +0000 UTC m=+2.864741091 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392370 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392361741 +0000 UTC m=+2.864756019 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392388 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392450 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392493 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392511 4870 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392533 4870 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.392537 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392572 4870 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392586 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.39256988 +0000 UTC m=+2.864964124 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392606 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392594612 +0000 UTC m=+2.864988855 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392630 4870 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392672 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs podName:b9dacaea-1664-495c-b164-a939a91bac87 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392661433 +0000 UTC m=+2.865055661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs") pod "network-metrics-daemon-qnzx5" (UID: "b9dacaea-1664-495c-b164-a939a91bac87") : secret "metrics-daemon-secret" not found Oct 31 07:29:17.392712 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392685 4870 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:17.393310 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.392765 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:18.392736748 +0000 UTC m=+2.865130988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:17.716089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.716022 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:29:17.716463 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.716389 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/1.log" Oct 31 07:29:17.716463 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.716418 4870 generic.go:356] "Generic (PLEG): container finished" podID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" containerID="afad7f39f9c62a2f5fe4e9899e3dbdf4634ec289785bbe5ce2f436234e2b8298" exitCode=255 Oct 31 07:29:17.716535 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.716493 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" event={"ID":"a4a619c0-1d45-4bfe-b948-f79f3d461cb2","Type":"ContainerDied","Data":"afad7f39f9c62a2f5fe4e9899e3dbdf4634ec289785bbe5ce2f436234e2b8298"} Oct 31 07:29:17.716565 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.716548 4870 scope.go:122] "RemoveContainer" containerID="0877a859b0f43add6a1f23219ac6b166e685e7227b69daf6afe214b08cc6a8c4" Oct 31 07:29:17.717015 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.716994 4870 scope.go:122] "RemoveContainer" containerID="afad7f39f9c62a2f5fe4e9899e3dbdf4634ec289785bbe5ce2f436234e2b8298" Oct 31 07:29:17.717230 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:17.717212 4870 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hzng6_openshift-console-operator(a4a619c0-1d45-4bfe-b948-f79f3d461cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" podUID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" Oct 31 07:29:17.811920 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.811887 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:17.988068 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:17.987986 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:18.128994 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.128958 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:18.402285 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402254 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:18.402443 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402294 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:18.402443 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402335 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:18.402443 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402354 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:18.402443 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402382 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:18.402443 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402417 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402400437 +0000 UTC m=+4.874794679 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402485 4870 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402512 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402519 4870 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402537 4870 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57d7689f6c-h8wmf: secret "image-registry-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402485 4870 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402538 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402525562 +0000 UTC m=+4.874919788 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402583 4870 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402607 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402599026 +0000 UTC m=+4.874993255 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402625 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402613083 +0000 UTC m=+4.875007323 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402643 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls podName:245742f2-7508-4558-93cf-42487bedc917 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.4026335 +0000 UTC m=+4.875027729 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls") pod "image-registry-57d7689f6c-h8wmf" (UID: "245742f2-7508-4558-93cf-42487bedc917") : secret "image-registry-tls" not found Oct 31 07:29:18.402669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402660 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402701 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402734 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402759 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402785 4870 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.402830 4870 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402838 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402830477 +0000 UTC m=+4.875224720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402871 4870 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402895 4870 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-546c4cd566-xfkx2: secret "image-registry-tls" not found Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402913 4870 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402936 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls podName:5ae4c917-214d-43eb-b14e-f58c5a8e755f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402926363 +0000 UTC m=+4.875320589 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls") pod "image-registry-546c4cd566-xfkx2" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f") : secret "image-registry-tls" not found Oct 31 07:29:18.403072 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:18.402953 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:20.402941723 +0000 UTC m=+4.875335977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:18.404913 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.404893 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b9dacaea-1664-495c-b164-a939a91bac87-metrics-certs\") pod \"network-metrics-daemon-qnzx5\" (UID: \"b9dacaea-1664-495c-b164-a939a91bac87\") " pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:18.405019 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.405006 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/92342503-edc9-45e9-b9d7-fff71ce5092d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-vq7hc\" (UID: \"92342503-edc9-45e9-b9d7-fff71ce5092d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:18.470531 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.470507 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" Oct 31 07:29:18.477201 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.477171 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qnzx5" Oct 31 07:29:18.721188 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.721108 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:29:18.781441 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.781411 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc"] Oct 31 07:29:18.784577 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:18.784552 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92342503_edc9_45e9_b9d7_fff71ce5092d.slice/crio-34cf133e170370688dfb539a05ef55e33b00b627b97369c48ae0a49e7f57ae3b WatchSource:0}: Error finding container 34cf133e170370688dfb539a05ef55e33b00b627b97369c48ae0a49e7f57ae3b: Status 404 returned error can't find the container with id 34cf133e170370688dfb539a05ef55e33b00b627b97369c48ae0a49e7f57ae3b Oct 31 07:29:18.786461 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.786440 4870 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 07:29:18.804965 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.804926 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qnzx5"] Oct 31 07:29:18.808211 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:18.808188 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9dacaea_1664_495c_b164_a939a91bac87.slice/crio-afdf5cec7ba1b18858a252da6fed4fe4eaf05460ed01fd56ffef98c59438e5aa WatchSource:0}: Error finding container afdf5cec7ba1b18858a252da6fed4fe4eaf05460ed01fd56ffef98c59438e5aa: Status 404 returned error can't find the container with id afdf5cec7ba1b18858a252da6fed4fe4eaf05460ed01fd56ffef98c59438e5aa Oct 31 07:29:18.936699 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:18.936666 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:19.335669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.335616 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:19.503112 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.503046 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:19.621668 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.621590 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:19.667262 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.667236 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sv7zk_a122982a-c398-47f1-a368-fd89db8899a7/dns-node-resolver/0.log" Oct 31 07:29:19.727192 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.727159 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnzx5" event={"ID":"b9dacaea-1664-495c-b164-a939a91bac87","Type":"ContainerStarted","Data":"afdf5cec7ba1b18858a252da6fed4fe4eaf05460ed01fd56ffef98c59438e5aa"} Oct 31 07:29:19.728806 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.728769 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" event={"ID":"92342503-edc9-45e9-b9d7-fff71ce5092d","Type":"ContainerStarted","Data":"34cf133e170370688dfb539a05ef55e33b00b627b97369c48ae0a49e7f57ae3b"} Oct 31 07:29:19.731056 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:19.731026 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:20.129605 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.129564 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:20.419270 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419191 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:20.419270 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419230 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:20.419270 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419252 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419280 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419312 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419332 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419347 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419356 4870 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419411 4870 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419425 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls podName:8039f97e-ddca-4960-80a9-b53c24950ca9 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:24.419405113 +0000 UTC m=+8.891799361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls") pod "dns-default-m99x4" (UID: "8039f97e-ddca-4960-80a9-b53c24950ca9") : secret "dns-default-metrics-tls" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419356 4870 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419447 4870 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419454 4870 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419452 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls podName:96548fcd-4945-49fd-a9d4-1577c47e0ed1 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:24.419440011 +0000 UTC m=+8.891834252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-9bqjc" (UID: "96548fcd-4945-49fd-a9d4-1577c47e0ed1") : secret "samples-operator-tls" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419499 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls podName:0e2c4370-aeba-4e68-ad8d-396664e91a83 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:24.419487913 +0000 UTC m=+8.891882139 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zdg8b" (UID: "0e2c4370-aeba-4e68-ad8d-396664e91a83") : secret "insights-runtime-extractor-tls" not found Oct 31 07:29:20.419536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.419511 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:20.419977 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419555 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:24.419548307 +0000 UTC m=+8.891942532 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : secret "router-metrics-certs-default" not found Oct 31 07:29:20.419977 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419566 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:24.419561075 +0000 UTC m=+8.891955301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:20.419977 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.419614 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle podName:af42539c-c267-4894-a291-764cbb70c30f nodeName:}" failed. No retries permitted until 2026-10-31 07:29:24.419605194 +0000 UTC m=+8.891999439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle") pod "router-default-5b4c78c6d5-pmxm4" (UID: "af42539c-c267-4894-a291-764cbb70c30f") : configmap references non-existent config key: service-ca.crt Oct 31 07:29:20.421738 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.421713 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"image-registry-546c4cd566-xfkx2\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:20.421841 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.421751 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"image-registry-57d7689f6c-h8wmf\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:20.573406 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.573376 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:20.579708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.579674 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:20.596279 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.596232 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:20.668810 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.668617 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m8msg_9e12a075-b4ec-44e3-8811-7b19f1410513/node-ca/0.log" Oct 31 07:29:20.733954 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.733886 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnzx5" event={"ID":"b9dacaea-1664-495c-b164-a939a91bac87","Type":"ContainerStarted","Data":"571d4545550f725a63e318414f29406cb5a965a3e6d447357f12a5d2f8786e48"} Oct 31 07:29:20.733954 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.733940 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qnzx5" event={"ID":"b9dacaea-1664-495c-b164-a939a91bac87","Type":"ContainerStarted","Data":"d500498b888e5de1c27615ca6b21c7c2f1d484bcaf89b5cb1dfaa45ca3b14408"} Oct 31 07:29:20.735442 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.735420 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-vq7hc" event={"ID":"92342503-edc9-45e9-b9d7-fff71ce5092d","Type":"ContainerStarted","Data":"45815c2efa245a24d5175227e0e1107ed79964c0b4a8715184a670dcdcb9a2d3"} Oct 31 07:29:20.888148 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.888114 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57d7689f6c-h8wmf"] Oct 31 07:29:20.892734 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:20.892706 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod245742f2_7508_4558_93cf_42487bedc917.slice/crio-66b18b641576a23fc46595bd634f04afaf898ee85c3177969a15ca703515db1e WatchSource:0}: Error finding container 66b18b641576a23fc46595bd634f04afaf898ee85c3177969a15ca703515db1e: Status 404 returned error can't find the container with id 66b18b641576a23fc46595bd634f04afaf898ee85c3177969a15ca703515db1e Oct 31 07:29:20.903022 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.902998 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:20.904883 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.904866 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-546c4cd566-xfkx2"] Oct 31 07:29:20.907816 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:20.907774 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae4c917_214d_43eb_b14e_f58c5a8e755f.slice/crio-6429a4fc6aa2405d8f487e89d53fea335efb37fd5a4e196485154481ea0c5308 WatchSource:0}: Error finding container 6429a4fc6aa2405d8f487e89d53fea335efb37fd5a4e196485154481ea0c5308: Status 404 returned error can't find the container with id 6429a4fc6aa2405d8f487e89d53fea335efb37fd5a4e196485154481ea0c5308 Oct 31 07:29:20.926098 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.926076 4870 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:20.926416 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:20.926399 4870 scope.go:122] "RemoveContainer" containerID="afad7f39f9c62a2f5fe4e9899e3dbdf4634ec289785bbe5ce2f436234e2b8298" Oct 31 07:29:20.926608 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:20.926588 4870 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hzng6_openshift-console-operator(a4a619c0-1d45-4bfe-b948-f79f3d461cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" podUID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" Oct 31 07:29:21.016357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.016272 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:21.069114 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.069080 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g29bz_e71bf711-37d2-421a-8bb5-a838f60d36dc/serve-healthcheck-canary/0.log" Oct 31 07:29:21.328336 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.328290 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:21.438479 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.438444 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:21.555824 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.555750 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:21.669286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.669156 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:21.740507 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.740470 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" event={"ID":"245742f2-7508-4558-93cf-42487bedc917","Type":"ContainerStarted","Data":"ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947"} Oct 31 07:29:21.740995 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.740517 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" event={"ID":"245742f2-7508-4558-93cf-42487bedc917","Type":"ContainerStarted","Data":"66b18b641576a23fc46595bd634f04afaf898ee85c3177969a15ca703515db1e"} Oct 31 07:29:21.741971 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.741943 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" event={"ID":"5ae4c917-214d-43eb-b14e-f58c5a8e755f","Type":"ContainerStarted","Data":"01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264"} Oct 31 07:29:21.742092 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.741987 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" event={"ID":"5ae4c917-214d-43eb-b14e-f58c5a8e755f","Type":"ContainerStarted","Data":"6429a4fc6aa2405d8f487e89d53fea335efb37fd5a4e196485154481ea0c5308"} Oct 31 07:29:21.781441 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.781413 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:21.917078 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:21.917048 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:22.072113 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:22.072083 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-njqtj_20fefbf1-a4ae-4208-a927-2dc47132a88c/kube-storage-version-migrator-operator/0.log" Oct 31 07:29:22.135524 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:22.135487 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:22.745142 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:22.745114 4870 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 07:29:22.745549 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:22.745114 4870 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 07:29:22.933105 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:22.933067 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:23.329000 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:23.328968 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:23.451308 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:23.451277 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:23.461175 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:23.461153 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:23.558624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:23.558590 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:23.737035 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:23.736941 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:24.133896 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.133857 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:24.459198 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.459111 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:24.459198 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.459167 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:24.459198 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.459193 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:24.459475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.459218 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:24.459475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.459242 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:24.459475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.459268 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:24.459475 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:24.459392 4870 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 31 07:29:24.459680 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:24.459486 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert podName:5fec1a19-aa94-423d-a573-71c8b0739f55 nodeName:}" failed. No retries permitted until 2026-10-31 07:29:32.459465463 +0000 UTC m=+16.931859692 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-p9pnl" (UID: "5fec1a19-aa94-423d-a573-71c8b0739f55") : secret "networking-console-plugin-cert" not found Oct 31 07:29:24.460213 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.460183 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af42539c-c267-4894-a291-764cbb70c30f-service-ca-bundle\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:24.461841 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.461815 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0e2c4370-aeba-4e68-ad8d-396664e91a83-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zdg8b\" (UID: \"0e2c4370-aeba-4e68-ad8d-396664e91a83\") " pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:24.461961 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.461887 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af42539c-c267-4894-a291-764cbb70c30f-metrics-certs\") pod \"router-default-5b4c78c6d5-pmxm4\" (UID: \"af42539c-c267-4894-a291-764cbb70c30f\") " pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:24.462101 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.462082 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8039f97e-ddca-4960-80a9-b53c24950ca9-metrics-tls\") pod \"dns-default-m99x4\" (UID: \"8039f97e-ddca-4960-80a9-b53c24950ca9\") " pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:24.462156 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.462113 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96548fcd-4945-49fd-a9d4-1577c47e0ed1-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-9bqjc\" (UID: \"96548fcd-4945-49fd-a9d4-1577c47e0ed1\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:24.472009 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.471986 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:24.472103 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.472075 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" Oct 31 07:29:24.484181 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.484160 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:24.484405 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.484379 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zdg8b" Oct 31 07:29:24.812534 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.812503 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc"] Oct 31 07:29:24.815635 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.815613 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m99x4"] Oct 31 07:29:24.816614 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.816482 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zdg8b"] Oct 31 07:29:24.819105 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:24.819077 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e2c4370_aeba_4e68_ad8d_396664e91a83.slice/crio-64199752e13fb6c50988309bea7c2ddb8f1f3e3213e1a3294e0e6ca4fdaab277 WatchSource:0}: Error finding container 64199752e13fb6c50988309bea7c2ddb8f1f3e3213e1a3294e0e6ca4fdaab277: Status 404 returned error can't find the container with id 64199752e13fb6c50988309bea7c2ddb8f1f3e3213e1a3294e0e6ca4fdaab277 Oct 31 07:29:24.824162 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.824133 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b4c78c6d5-pmxm4"] Oct 31 07:29:24.833135 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:24.833069 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf42539c_c267_4894_a291_764cbb70c30f.slice/crio-76c088ac1c6f0a4328602cf1d2980e3ee9a566b368d3e1eb261d83435b563d4f WatchSource:0}: Error finding container 76c088ac1c6f0a4328602cf1d2980e3ee9a566b368d3e1eb261d83435b563d4f: Status 404 returned error can't find the container with id 76c088ac1c6f0a4328602cf1d2980e3ee9a566b368d3e1eb261d83435b563d4f Oct 31 07:29:24.932552 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:24.932518 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:25.316515 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.316480 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:25.347932 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.347899 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:25.348172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.348156 4870 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 07:29:25.368197 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.368164 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:25.427212 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.427172 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:25.535122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.535096 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:25.689347 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.689271 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:25.735363 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.735297 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:25.760113 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.760077 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" event={"ID":"96548fcd-4945-49fd-a9d4-1577c47e0ed1","Type":"ContainerStarted","Data":"4c7eb435d1d2269988a7641405ba94b401b71e6a50713800653af5ed595dc155"} Oct 31 07:29:25.762079 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.762007 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" event={"ID":"af42539c-c267-4894-a291-764cbb70c30f","Type":"ContainerStarted","Data":"7a15014442db4f01dc1f7fca37e35c76200e625d36fc61de24cfab4f4894a6f4"} Oct 31 07:29:25.762079 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.762050 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" event={"ID":"af42539c-c267-4894-a291-764cbb70c30f","Type":"ContainerStarted","Data":"76c088ac1c6f0a4328602cf1d2980e3ee9a566b368d3e1eb261d83435b563d4f"} Oct 31 07:29:25.763350 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.763321 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m99x4" event={"ID":"8039f97e-ddca-4960-80a9-b53c24950ca9","Type":"ContainerStarted","Data":"5c5dfe28021159c03b06271862e38520191bd8149d957013122ce2289df935ca"} Oct 31 07:29:25.765190 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.765162 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdg8b" event={"ID":"0e2c4370-aeba-4e68-ad8d-396664e91a83","Type":"ContainerStarted","Data":"b674b7fb8db907dfc4cc3b24099772e421daca743e0d964865a70f36b827ae3f"} Oct 31 07:29:25.765295 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.765206 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdg8b" event={"ID":"0e2c4370-aeba-4e68-ad8d-396664e91a83","Type":"ContainerStarted","Data":"4c2fcfba957ae1911cea8b75f121719eac5956a873750500a6d397eb55b04069"} Oct 31 07:29:25.765295 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:25.765232 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdg8b" event={"ID":"0e2c4370-aeba-4e68-ad8d-396664e91a83","Type":"ContainerStarted","Data":"64199752e13fb6c50988309bea7c2ddb8f1f3e3213e1a3294e0e6ca4fdaab277"} Oct 31 07:29:26.131369 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.131317 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:26.132118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.132097 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:26.132604 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.132588 4870 scope.go:122] "RemoveContainer" containerID="afad7f39f9c62a2f5fe4e9899e3dbdf4634ec289785bbe5ce2f436234e2b8298" Oct 31 07:29:26.132919 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:26.132893 4870 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-hzng6_openshift-console-operator(a4a619c0-1d45-4bfe-b948-f79f3d461cb2)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" podUID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" Oct 31 07:29:26.358761 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.358733 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:26.472670 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.472589 4870 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:26.475475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.475440 4870 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:26.771947 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.771910 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" event={"ID":"96548fcd-4945-49fd-a9d4-1577c47e0ed1","Type":"ContainerStarted","Data":"b3d63e6380f73f6cdec240d1ce35b2ebaa5044293f904ed3b9303015c847203b"} Oct 31 07:29:26.772091 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.771979 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-9bqjc" event={"ID":"96548fcd-4945-49fd-a9d4-1577c47e0ed1","Type":"ContainerStarted","Data":"1730d20d735fdca94b2f5df2fcb18c278636c4b5d10897d56542f98baa9ead3b"} Oct 31 07:29:26.775603 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.775579 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m99x4" event={"ID":"8039f97e-ddca-4960-80a9-b53c24950ca9","Type":"ContainerStarted","Data":"836d4ca0af22945f354a8451211886d6dee7b4d1b5e0237e7e5265e30b5713ba"} Oct 31 07:29:26.775697 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.775623 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:26.775697 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.775639 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m99x4" event={"ID":"8039f97e-ddca-4960-80a9-b53c24950ca9","Type":"ContainerStarted","Data":"4e5ec14356c6db6dc3d2b922a94c30ef184065e80913e1636444cf9a01be0178"} Oct 31 07:29:26.937964 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:26.937935 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:27.330632 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.330597 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:27.530553 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.530521 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:27.656708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.656629 4870 scope.go:122] "RemoveContainer" containerID="afad7f39f9c62a2f5fe4e9899e3dbdf4634ec289785bbe5ce2f436234e2b8298" Oct 31 07:29:27.730238 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.730214 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:27.780497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.780461 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zdg8b" event={"ID":"0e2c4370-aeba-4e68-ad8d-396664e91a83","Type":"ContainerStarted","Data":"f0fc4971e5bea8ea34bf289535b3818ff2700c176dab397f9b02497452233d2e"} Oct 31 07:29:27.782298 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.782280 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:29:27.782420 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.782399 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" event={"ID":"a4a619c0-1d45-4bfe-b948-f79f3d461cb2","Type":"ContainerStarted","Data":"23caec8e3e8ffebe6d990fcdcbbcc63a9eda58047de6083bc73926172b3369a3"} Oct 31 07:29:27.782574 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.782560 4870 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 07:29:27.783129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.783112 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:27.784130 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.784107 4870 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-hzng6 container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" start-of-body= Oct 31 07:29:27.784216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:27.784148 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" podUID="a4a619c0-1d45-4bfe-b948-f79f3d461cb2" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.20:8443/readyz\": dial tcp 10.132.0.20:8443: connect: connection refused" Oct 31 07:29:28.132376 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.132338 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:28.268863 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.268817 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2qrrp" Oct 31 07:29:28.354107 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.354080 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:28.465746 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.465670 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:28.574715 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.574683 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:28.732069 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.731992 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:28.789677 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:28.789651 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-hzng6" Oct 31 07:29:29.103461 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:29.103428 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:29.220563 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:29.220530 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:29.725058 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:29.725025 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:29.836624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:29.836592 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:30.163153 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.163118 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:30.163330 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.163254 4870 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 07:29:30.164338 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.164314 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b4c78c6d5-pmxm4" Oct 31 07:29:30.593757 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.593726 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57d7689f6c-h8wmf"] Oct 31 07:29:30.710827 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.710787 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:30.831683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.831652 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:30.946590 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:30.946518 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:31.597805 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:31.597764 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2b2kf" Oct 31 07:29:32.526589 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:32.526551 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:32.529020 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:32.528998 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5fec1a19-aa94-423d-a573-71c8b0739f55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-p9pnl\" (UID: \"5fec1a19-aa94-423d-a573-71c8b0739f55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:32.573706 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:32.573673 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" Oct 31 07:29:32.865174 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:32.865142 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-p9pnl"] Oct 31 07:29:32.868350 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:32.868323 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fec1a19_aa94_423d_a573_71c8b0739f55.slice/crio-166bd9ac4a8ffb5698f91f6ace26e684b4f92a500119c2187581e68206cf89d2 WatchSource:0}: Error finding container 166bd9ac4a8ffb5698f91f6ace26e684b4f92a500119c2187581e68206cf89d2: Status 404 returned error can't find the container with id 166bd9ac4a8ffb5698f91f6ace26e684b4f92a500119c2187581e68206cf89d2 Oct 31 07:29:33.805402 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:33.805269 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" event={"ID":"5fec1a19-aa94-423d-a573-71c8b0739f55","Type":"ContainerStarted","Data":"166bd9ac4a8ffb5698f91f6ace26e684b4f92a500119c2187581e68206cf89d2"} Oct 31 07:29:34.417974 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:34.417913 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:34.529611 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:34.529575 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:34.638129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:34.638082 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:34.810307 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:34.810271 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-p9pnl" event={"ID":"5fec1a19-aa94-423d-a573-71c8b0739f55","Type":"ContainerStarted","Data":"664c67b52fa518c487e5cd7da79083718104aa6b6f41cf530389b0c05795eac5"} Oct 31 07:29:35.693830 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:35.693776 4870 patch_prober.go:29] interesting pod/image-registry-546c4cd566-xfkx2 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 07:29:35.693988 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:35.693845 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 07:29:36.095626 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:36.095590 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:29:36.605868 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:36.605828 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:36.713468 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:36.713425 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:36.785390 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:36.785365 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m99x4" Oct 31 07:29:37.122450 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:37.122415 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:38.897188 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:38.897153 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.052215 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.052185 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.177558 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.177446 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.225175 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.225139 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vxgj7"] Oct 31 07:29:39.229993 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.229972 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.251836 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.251806 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b5fc6\"" Oct 31 07:29:39.251836 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.251810 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 07:29:39.251977 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.251870 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 07:29:39.277130 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277103 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-textfile\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277231 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277137 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e2536a-e888-4f15-9d91-52e6cd67aae2-metrics-client-ca\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277231 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277170 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-accelerators-collector-config\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277307 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277259 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-sys\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277307 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277277 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-tls\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277307 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277293 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277395 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277315 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-wtmp\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277427 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-root\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.277474 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.277450 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twkbd\" (UniqueName: \"kubernetes.io/projected/31e2536a-e888-4f15-9d91-52e6cd67aae2-kube-api-access-twkbd\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.295689 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.295668 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 07:29:39.301732 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.301713 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 07:29:39.305922 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.305865 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.378438 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378412 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-textfile\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.378549 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378445 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e2536a-e888-4f15-9d91-52e6cd67aae2-metrics-client-ca\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.378549 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378471 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-accelerators-collector-config\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.378761 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378737 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-sys\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.378849 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378780 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-tls\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.378849 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378824 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-textfile\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.378849 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378831 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379005 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378842 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-sys\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379005 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378894 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-wtmp\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379005 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378927 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-root\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379005 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.378946 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twkbd\" (UniqueName: \"kubernetes.io/projected/31e2536a-e888-4f15-9d91-52e6cd67aae2-kube-api-access-twkbd\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379213 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.379079 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-wtmp\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379213 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.379137 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31e2536a-e888-4f15-9d91-52e6cd67aae2-root\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379213 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.379196 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-accelerators-collector-config\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.379336 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.379195 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31e2536a-e888-4f15-9d91-52e6cd67aae2-metrics-client-ca\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.381094 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.381077 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-tls\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.381167 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.381151 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31e2536a-e888-4f15-9d91-52e6cd67aae2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.400299 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.400278 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twkbd\" (UniqueName: \"kubernetes.io/projected/31e2536a-e888-4f15-9d91-52e6cd67aae2-kube-api-access-twkbd\") pod \"node-exporter-vxgj7\" (UID: \"31e2536a-e888-4f15-9d91-52e6cd67aae2\") " pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.417730 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.417708 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.531876 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.531770 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.545655 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.545621 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vxgj7" Oct 31 07:29:39.559601 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:29:39.559575 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e2536a_e888_4f15_9d91_52e6cd67aae2.slice/crio-f44da7f7bc22d3595851159b147ece7915fe689b0949c283c0fd536f9b5f3b10 WatchSource:0}: Error finding container f44da7f7bc22d3595851159b147ece7915fe689b0949c283c0fd536f9b5f3b10: Status 404 returned error can't find the container with id f44da7f7bc22d3595851159b147ece7915fe689b0949c283c0fd536f9b5f3b10 Oct 31 07:29:39.643327 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.643296 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:39.827762 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:39.827731 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vxgj7" event={"ID":"31e2536a-e888-4f15-9d91-52e6cd67aae2","Type":"ContainerStarted","Data":"f44da7f7bc22d3595851159b147ece7915fe689b0949c283c0fd536f9b5f3b10"} Oct 31 07:29:40.281536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:40.281454 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:40.580048 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:40.580010 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:40.598887 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:40.598858 4870 patch_prober.go:29] interesting pod/image-registry-57d7689f6c-h8wmf container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 07:29:40.599023 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:40.598907 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" podUID="245742f2-7508-4558-93cf-42487bedc917" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 07:29:40.692438 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:40.692372 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:41.264397 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:41.264316 4870 patch_prober.go:29] interesting pod/image-registry-546c4cd566-xfkx2 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 31 07:29:41.264397 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:41.264376 4870 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 07:29:41.836282 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:41.836243 4870 generic.go:356] "Generic (PLEG): container finished" podID="31e2536a-e888-4f15-9d91-52e6cd67aae2" containerID="f790cacb1e4636adf87c0d9c824d36d00a4dbcb611c453949f00d868ccf43caa" exitCode=0 Oct 31 07:29:41.836635 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:41.836304 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vxgj7" event={"ID":"31e2536a-e888-4f15-9d91-52e6cd67aae2","Type":"ContainerDied","Data":"f790cacb1e4636adf87c0d9c824d36d00a4dbcb611c453949f00d868ccf43caa"} Oct 31 07:29:41.967331 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:41.967310 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:42.082960 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.082929 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:42.193999 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.193915 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:42.485769 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.485691 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:42.607266 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.607239 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:42.716107 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.716075 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:42.842082 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.842047 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vxgj7" event={"ID":"31e2536a-e888-4f15-9d91-52e6cd67aae2","Type":"ContainerStarted","Data":"b9c636b0ff05fd39f982c291790a49f7204022cb6492176a667b42ff4dec63e6"} Oct 31 07:29:42.842443 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.842116 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vxgj7" event={"ID":"31e2536a-e888-4f15-9d91-52e6cd67aae2","Type":"ContainerStarted","Data":"ab631cfa328a6d5b3fa76b35993ef821d01720fff0c94cf99d4a93ed71c23131"} Oct 31 07:29:42.906047 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:42.906003 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vxgj7" podStartSLOduration=2.559854124 podStartE2EDuration="3.90599039s" totalImagesPullingTime="1.266024595s" totalInitContainerRuntime="80.111671ms" isStatefulPod=true podCreationTimestamp="2026-10-31 07:29:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:29:42.89415191 +0000 UTC m=+27.366546152" watchObservedRunningTime="2026-10-31 07:29:42.90599039 +0000 UTC m=+27.378384637" Oct 31 07:29:43.130286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:43.130193 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:43.252816 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:43.252749 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:43.464412 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:43.464329 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gdgxp" Oct 31 07:29:43.998739 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:43.998701 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:44.134081 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:44.134048 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:44.297805 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:44.297760 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:45.140171 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:45.140137 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:45.254085 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:45.254052 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:45.693590 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:45.693560 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:29:45.975624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:45.975539 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:46.090815 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:46.090761 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:46.226360 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:46.226263 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:50.169431 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:50.169398 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:50.279191 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:50.279156 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:50.597844 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:50.597818 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:51.633417 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:51.633379 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:51.743531 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:51.743498 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:52.578214 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:52.578177 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546c4cd566-xfkx2"] Oct 31 07:29:52.684766 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:52.684722 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:52.793130 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:52.793098 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:52.903905 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:52.903815 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:55.615236 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.615199 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" podUID="245742f2-7508-4558-93cf-42487bedc917" containerName="registry" containerID="cri-o://ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947" gracePeriod=30 Oct 31 07:29:55.854294 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.854272 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:55.893783 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.893719 4870 generic.go:356] "Generic (PLEG): container finished" podID="245742f2-7508-4558-93cf-42487bedc917" containerID="ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947" exitCode=0 Oct 31 07:29:55.893783 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.893763 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" Oct 31 07:29:55.893945 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.893823 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" event={"ID":"245742f2-7508-4558-93cf-42487bedc917","Type":"ContainerDied","Data":"ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947"} Oct 31 07:29:55.893945 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.893874 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57d7689f6c-h8wmf" event={"ID":"245742f2-7508-4558-93cf-42487bedc917","Type":"ContainerDied","Data":"66b18b641576a23fc46595bd634f04afaf898ee85c3177969a15ca703515db1e"} Oct 31 07:29:55.893945 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.893901 4870 scope.go:122] "RemoveContainer" containerID="ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947" Oct 31 07:29:55.902054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.902032 4870 scope.go:122] "RemoveContainer" containerID="ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947" Oct 31 07:29:55.902367 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:29:55.902348 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947\": container with ID starting with ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947 not found: ID does not exist" containerID="ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947" Oct 31 07:29:55.902420 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.902376 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947"} err="failed to get container status \"ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947\": rpc error: code = NotFound desc = could not find container \"ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947\": container with ID starting with ec623b1c43b8cb3870126cf7f480f79405f106fa7b87a8b77c3e20068bf2f947 not found: ID does not exist" Oct 31 07:29:55.917043 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917015 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917157 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917094 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917157 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917151 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917191 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917244 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917273 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917321 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917348 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") pod \"245742f2-7508-4558-93cf-42487bedc917\" (UID: \"245742f2-7508-4558-93cf-42487bedc917\") " Oct 31 07:29:55.917590 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917567 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:29:55.917977 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.917953 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:29:55.920025 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.919999 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:29:55.920191 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.920164 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:29:55.920260 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.920211 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "kube-api-access-d9jxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:29:55.920917 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.920893 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:29:55.921012 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.920961 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:29:55.926220 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:55.926196 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted" pod "245742f2-7508-4558-93cf-42487bedc917" (UID: "245742f2-7508-4558-93cf-42487bedc917"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:29:56.018391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018356 4870 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-registry-certificates\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018385 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9jxl\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-kube-api-access-d9jxl\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018395 4870 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-installation-pull-secrets\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018607 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018405 4870 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/245742f2-7508-4558-93cf-42487bedc917-image-registry-private-configuration\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018607 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018416 4870 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/245742f2-7508-4558-93cf-42487bedc917-trusted-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018607 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018424 4870 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-registry-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018607 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018432 4870 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/245742f2-7508-4558-93cf-42487bedc917-bound-sa-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.018607 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.018439 4870 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/245742f2-7508-4558-93cf-42487bedc917-ca-trust-extracted\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:29:56.236731 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.236699 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57d7689f6c-h8wmf"] Oct 31 07:29:56.248113 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:56.248090 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57d7689f6c-h8wmf"] Oct 31 07:29:57.661421 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:57.661347 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="245742f2-7508-4558-93cf-42487bedc917" path="/var/lib/kubelet/pods/245742f2-7508-4558-93cf-42487bedc917/volumes" Oct 31 07:29:57.939593 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:57.939511 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:58.058464 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:58.058431 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:29:58.937980 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:29:58.937943 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:00.173183 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:00.173151 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:00.282581 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:00.282550 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:00.659227 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:00.659188 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:00.768386 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:00.768327 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:06.096534 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:30:06.096487 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:30:06.317335 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:06.317300 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:06.466373 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:06.466283 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:06.602107 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:06.602068 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:06.712367 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:06.712325 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:17.597549 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.597476 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerName="registry" containerID="cri-o://01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264" gracePeriod=30 Oct 31 07:30:17.839737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.839711 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:30:17.978337 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.978247 4870 generic.go:356] "Generic (PLEG): container finished" podID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerID="01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264" exitCode=0 Oct 31 07:30:17.978337 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.978295 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" Oct 31 07:30:17.978511 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.978327 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" event={"ID":"5ae4c917-214d-43eb-b14e-f58c5a8e755f","Type":"ContainerDied","Data":"01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264"} Oct 31 07:30:17.978511 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.978382 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-546c4cd566-xfkx2" event={"ID":"5ae4c917-214d-43eb-b14e-f58c5a8e755f","Type":"ContainerDied","Data":"6429a4fc6aa2405d8f487e89d53fea335efb37fd5a4e196485154481ea0c5308"} Oct 31 07:30:17.978511 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.978405 4870 scope.go:122] "RemoveContainer" containerID="01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264" Oct 31 07:30:17.986203 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.986186 4870 scope.go:122] "RemoveContainer" containerID="01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264" Oct 31 07:30:17.986465 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:30:17.986443 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264\": container with ID starting with 01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264 not found: ID does not exist" containerID="01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264" Oct 31 07:30:17.986557 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.986474 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264"} err="failed to get container status \"01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264\": rpc error: code = NotFound desc = could not find container \"01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264\": container with ID starting with 01e159282dd89f4ea2fbc666c46dc928d6666b57800ba342b54b326fef74c264 not found: ID does not exist" Oct 31 07:30:17.996443 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996423 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996464 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996495 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996529 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996726 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996585 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996726 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996606 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996726 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996638 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996726 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996687 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") pod \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\" (UID: \"5ae4c917-214d-43eb-b14e-f58c5a8e755f\") " Oct 31 07:30:17.996986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.996965 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:30:17.997041 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.997001 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:30:17.999146 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.999111 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "kube-api-access-92hm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:30:17.999268 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.999151 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:30:17.999322 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.999270 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:30:17.999322 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.999297 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:30:17.999395 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:17.999320 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:30:18.003562 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.003538 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted" pod "5ae4c917-214d-43eb-b14e-f58c5a8e755f" (UID: "5ae4c917-214d-43eb-b14e-f58c5a8e755f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:30:18.098185 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098141 4870 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-bound-sa-token\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098185 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098180 4870 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5ae4c917-214d-43eb-b14e-f58c5a8e755f-ca-trust-extracted\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098185 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098191 4870 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098185 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098200 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92hm9\" (UniqueName: \"kubernetes.io/projected/5ae4c917-214d-43eb-b14e-f58c5a8e755f-kube-api-access-92hm9\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098447 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098210 4870 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-installation-pull-secrets\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098447 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098219 4870 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5ae4c917-214d-43eb-b14e-f58c5a8e755f-image-registry-private-configuration\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098447 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098228 4870 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-trusted-ca\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.098447 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.098236 4870 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5ae4c917-214d-43eb-b14e-f58c5a8e755f-registry-certificates\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:30:18.320210 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.320180 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-546c4cd566-xfkx2"] Oct 31 07:30:18.332129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:18.332107 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-546c4cd566-xfkx2"] Oct 31 07:30:19.660750 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:19.660718 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" path="/var/lib/kubelet/pods/5ae4c917-214d-43eb-b14e-f58c5a8e755f/volumes" Oct 31 07:30:21.599192 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:21.599143 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:21.703440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:21.703408 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:24.973046 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:24.973016 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:25.077069 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:25.077040 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:28.619096 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:28.619056 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:28.727377 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:28.727336 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:30.224588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:30.224553 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:30.329645 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:30.329608 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:36.095092 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:30:36.095044 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:30:37.044959 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:37.044925 4870 generic.go:356] "Generic (PLEG): container finished" podID="20fefbf1-a4ae-4208-a927-2dc47132a88c" containerID="50f7cd999db68fd915663a71487b9e05e7287bc9699ed12821c54f7f32a3bc3c" exitCode=0 Oct 31 07:30:37.045123 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:37.044999 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" event={"ID":"20fefbf1-a4ae-4208-a927-2dc47132a88c","Type":"ContainerDied","Data":"50f7cd999db68fd915663a71487b9e05e7287bc9699ed12821c54f7f32a3bc3c"} Oct 31 07:30:37.045322 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:37.045303 4870 scope.go:122] "RemoveContainer" containerID="50f7cd999db68fd915663a71487b9e05e7287bc9699ed12821c54f7f32a3bc3c" Oct 31 07:30:38.050498 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:38.050459 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-njqtj" event={"ID":"20fefbf1-a4ae-4208-a927-2dc47132a88c","Type":"ContainerStarted","Data":"f44a838e66ad0f046a3dce1fb205cd62d308f2f3cf6560ad5a1db8c03868cff4"} Oct 31 07:30:38.435126 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:38.435026 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:38.538637 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:38.538603 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:42.066153 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:42.066120 4870 generic.go:356] "Generic (PLEG): container finished" podID="805181c8-c83e-433d-9980-62f629c63f0d" containerID="b32826ef5a483d8eb48dab8d6a9a8fc08deb090a89970f8cf2a4eea407f8acaf" exitCode=0 Oct 31 07:30:42.066530 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:42.066194 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" event={"ID":"805181c8-c83e-433d-9980-62f629c63f0d","Type":"ContainerDied","Data":"b32826ef5a483d8eb48dab8d6a9a8fc08deb090a89970f8cf2a4eea407f8acaf"} Oct 31 07:30:42.066530 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:42.066518 4870 scope.go:122] "RemoveContainer" containerID="b32826ef5a483d8eb48dab8d6a9a8fc08deb090a89970f8cf2a4eea407f8acaf" Oct 31 07:30:42.705117 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:42.705088 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:42.809030 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:42.809005 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:43.071739 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:43.071707 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-lrjlk" event={"ID":"805181c8-c83e-433d-9980-62f629c63f0d","Type":"ContainerStarted","Data":"2931f77d99a3de8e3a3cecf8b2c960f7e434d04a25aba88f8166e4e8eee160f2"} Oct 31 07:30:51.606878 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:51.606842 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:51.712990 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:51.712948 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:51.814867 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:51.814823 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:30:52.110477 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.110444 4870 generic.go:356] "Generic (PLEG): container finished" podID="8f52e908-ef31-4672-ac4d-1ee9778d1b52" containerID="405d214e96a6aee2a8dac5e60c14440585b71672dabb1c37fe067678c1f2b127" exitCode=0 Oct 31 07:30:52.110646 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.110516 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" event={"ID":"8f52e908-ef31-4672-ac4d-1ee9778d1b52","Type":"ContainerDied","Data":"405d214e96a6aee2a8dac5e60c14440585b71672dabb1c37fe067678c1f2b127"} Oct 31 07:30:52.110868 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.110854 4870 scope.go:122] "RemoveContainer" containerID="405d214e96a6aee2a8dac5e60c14440585b71672dabb1c37fe067678c1f2b127" Oct 31 07:30:52.771112 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771081 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-z54zk-w4l76"] Oct 31 07:30:52.771584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771365 4870 cpu_manager.go:414] "containerMap: removing container" podUID="245742f2-7508-4558-93cf-42487bedc917" containerName="registry" Oct 31 07:30:52.771584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771379 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="245742f2-7508-4558-93cf-42487bedc917" containerName="registry" Oct 31 07:30:52.771584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771394 4870 cpu_manager.go:414] "containerMap: removing container" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerName="registry" Oct 31 07:30:52.771584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771401 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerName="registry" Oct 31 07:30:52.771584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771493 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="245742f2-7508-4558-93cf-42487bedc917" containerName="registry" Oct 31 07:30:52.771584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.771503 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae4c917-214d-43eb-b14e-f58c5a8e755f" containerName="registry" Oct 31 07:30:52.774251 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.774234 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.799419 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.799394 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dl9sg\"" Oct 31 07:30:52.800467 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.800442 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-z54zk-w4l76"] Oct 31 07:30:52.859815 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.859773 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.859940 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.859824 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.859940 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.859846 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qll44\" (UniqueName: \"kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.860014 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.859958 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.961101 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.961067 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.961244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.961105 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qll44\" (UniqueName: \"kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.961244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.961136 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.961244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.961178 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.961637 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.961613 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.961756 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.961739 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.962084 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.962067 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:52.984068 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:52.984034 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qll44\" (UniqueName: \"kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44\") pod \"periodic-gathering-z54zk-w4l76\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:53.085698 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:53.085655 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:30:53.116130 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:53.116099 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-vvm6c" event={"ID":"8f52e908-ef31-4672-ac4d-1ee9778d1b52","Type":"ContainerStarted","Data":"0e5f248a6574fc3235f7fa0bf6f05fa367016d7043cf6a4b0bdc47f070aa6de7"} Oct 31 07:30:53.378606 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:53.378526 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-z54zk-w4l76"] Oct 31 07:30:53.381372 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:30:53.381334 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143aebad_6cd8_46bd_ac09_af425fe81be7.slice/crio-2bef4e9abbe9e4afe6b3afd2a97b9ebe36ef52f60c4c3b3cebab0bc00cad7190 WatchSource:0}: Error finding container 2bef4e9abbe9e4afe6b3afd2a97b9ebe36ef52f60c4c3b3cebab0bc00cad7190: Status 404 returned error can't find the container with id 2bef4e9abbe9e4afe6b3afd2a97b9ebe36ef52f60c4c3b3cebab0bc00cad7190 Oct 31 07:30:54.121123 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:54.121082 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z54zk-w4l76" event={"ID":"143aebad-6cd8-46bd-ac09-af425fe81be7","Type":"ContainerStarted","Data":"2cfb0e9c2e69533c86a75c7e66376c89c22406c9f88c8459bb5470631cef84fd"} Oct 31 07:30:54.121538 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:54.121148 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z54zk-w4l76" event={"ID":"143aebad-6cd8-46bd-ac09-af425fe81be7","Type":"ContainerStarted","Data":"2bef4e9abbe9e4afe6b3afd2a97b9ebe36ef52f60c4c3b3cebab0bc00cad7190"} Oct 31 07:30:54.159455 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:30:54.159405 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-z54zk-w4l76" podStartSLOduration=2.159393053 podStartE2EDuration="2.159393053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:30:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:30:54.159225015 +0000 UTC m=+98.631619266" watchObservedRunningTime="2026-10-31 07:30:54.159393053 +0000 UTC m=+98.631787295" Oct 31 07:31:06.095671 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:31:06.095631 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:31:06.391019 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:31:06.390916 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:31:28.282328 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:31:28.282259 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:31:36.095374 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:31:36.095327 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:32:06.094901 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:32:06.094857 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:32:06.522144 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:06.522071 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:17.412261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:17.412223 4870 generic.go:356] "Generic (PLEG): container finished" podID="143aebad-6cd8-46bd-ac09-af425fe81be7" containerID="2cfb0e9c2e69533c86a75c7e66376c89c22406c9f88c8459bb5470631cef84fd" exitCode=0 Oct 31 07:32:17.412729 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:17.412299 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z54zk-w4l76" event={"ID":"143aebad-6cd8-46bd-ac09-af425fe81be7","Type":"ContainerDied","Data":"2cfb0e9c2e69533c86a75c7e66376c89c22406c9f88c8459bb5470631cef84fd"} Oct 31 07:32:18.543241 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.543214 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:32:18.615036 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615002 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44\" (UniqueName: \"kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44\") pod \"143aebad-6cd8-46bd-ac09-af425fe81be7\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " Oct 31 07:32:18.615195 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615050 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle\") pod \"143aebad-6cd8-46bd-ac09-af425fe81be7\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " Oct 31 07:32:18.615195 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615082 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle\") pod \"143aebad-6cd8-46bd-ac09-af425fe81be7\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " Oct 31 07:32:18.615280 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615255 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path\") pod \"143aebad-6cd8-46bd-ac09-af425fe81be7\" (UID: \"143aebad-6cd8-46bd-ac09-af425fe81be7\") " Oct 31 07:32:18.615432 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615414 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle" pod "143aebad-6cd8-46bd-ac09-af425fe81be7" (UID: "143aebad-6cd8-46bd-ac09-af425fe81be7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:32:18.615535 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615514 4870 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-trusted-ca-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:32:18.615535 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615523 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle" pod "143aebad-6cd8-46bd-ac09-af425fe81be7" (UID: "143aebad-6cd8-46bd-ac09-af425fe81be7"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:32:18.615631 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.615565 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path" pod "143aebad-6cd8-46bd-ac09-af425fe81be7" (UID: "143aebad-6cd8-46bd-ac09-af425fe81be7"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:32:18.617253 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.617230 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44" pod "143aebad-6cd8-46bd-ac09-af425fe81be7" (UID: "143aebad-6cd8-46bd-ac09-af425fe81be7"). InnerVolumeSpecName "kube-api-access-qll44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:32:18.716630 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.716554 4870 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/143aebad-6cd8-46bd-ac09-af425fe81be7-archives-path\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:32:18.716630 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.716583 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qll44\" (UniqueName: \"kubernetes.io/projected/143aebad-6cd8-46bd-ac09-af425fe81be7-kube-api-access-qll44\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:32:18.716630 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:18.716593 4870 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/143aebad-6cd8-46bd-ac09-af425fe81be7-service-ca-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:32:19.421244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:19.421208 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z54zk-w4l76" event={"ID":"143aebad-6cd8-46bd-ac09-af425fe81be7","Type":"ContainerDied","Data":"2bef4e9abbe9e4afe6b3afd2a97b9ebe36ef52f60c4c3b3cebab0bc00cad7190"} Oct 31 07:32:19.421414 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:19.421260 4870 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bef4e9abbe9e4afe6b3afd2a97b9ebe36ef52f60c4c3b3cebab0bc00cad7190" Oct 31 07:32:19.421414 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:19.421221 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z54zk-w4l76" Oct 31 07:32:31.624243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:31.624212 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:36.095987 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:32:36.095952 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:32:46.594866 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:46.594828 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:46.697125 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:46.697085 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:46.800003 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:46.799974 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:47.827441 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.827403 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6"] Oct 31 07:32:47.827928 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.827854 4870 cpu_manager.go:414] "containerMap: removing container" podUID="143aebad-6cd8-46bd-ac09-af425fe81be7" containerName="insights-gathering" Oct 31 07:32:47.827928 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.827873 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143aebad-6cd8-46bd-ac09-af425fe81be7" containerName="insights-gathering" Oct 31 07:32:47.828099 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.828082 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="143aebad-6cd8-46bd-ac09-af425fe81be7" containerName="insights-gathering" Oct 31 07:32:47.831121 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.831099 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:47.845383 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.845356 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 31 07:32:47.856547 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.856524 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 31 07:32:47.856721 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.856706 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"98a21a31f7a8ae0212b9c22cc643f9202bfa37-6b211c7f-dockercfg-5jvln\"" Oct 31 07:32:47.857650 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.857630 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6"] Oct 31 07:32:47.945229 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.945192 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:47.945401 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.945239 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:47.945401 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:47.945318 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnn7p\" (UniqueName: \"kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.045909 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.045875 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnn7p\" (UniqueName: \"kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.046080 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.045936 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.046080 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.045958 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.046270 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.046256 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.046310 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.046283 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.068408 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.068381 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnn7p\" (UniqueName: \"kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p\") pod \"98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.146609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.146521 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:32:48.438111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.438034 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6"] Oct 31 07:32:48.441393 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:32:48.441366 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c75a9c_90e1_4258_9c80_6ca14db2133c.slice/crio-d43bebe42cfa28232ca78afa84ed2607bcc1dbe8b72a00454e5daa005a9ac222 WatchSource:0}: Error finding container d43bebe42cfa28232ca78afa84ed2607bcc1dbe8b72a00454e5daa005a9ac222: Status 404 returned error can't find the container with id d43bebe42cfa28232ca78afa84ed2607bcc1dbe8b72a00454e5daa005a9ac222 Oct 31 07:32:48.519530 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:48.519500 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerStarted","Data":"d43bebe42cfa28232ca78afa84ed2607bcc1dbe8b72a00454e5daa005a9ac222"} Oct 31 07:32:52.811114 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:52.811076 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:52.927743 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:52.927707 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:32:54.544644 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:54.544611 4870 generic.go:356] "Generic (PLEG): container finished" podID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerID="e99b57cfeffebff1aa0f647cf28ecdfe52b13d8d6fb3701c3bf9f62b373a108a" exitCode=0 Oct 31 07:32:54.545055 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:54.544688 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerDied","Data":"e99b57cfeffebff1aa0f647cf28ecdfe52b13d8d6fb3701c3bf9f62b373a108a"} Oct 31 07:32:56.561009 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:56.560971 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerStarted","Data":"eefbb921df91cd4467ae6b9e5f89bbec09ca198ec38639dae6558d30d8e42a54"} Oct 31 07:32:57.566201 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:57.566115 4870 generic.go:356] "Generic (PLEG): container finished" podID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerID="eefbb921df91cd4467ae6b9e5f89bbec09ca198ec38639dae6558d30d8e42a54" exitCode=0 Oct 31 07:32:57.566623 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:32:57.566204 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerDied","Data":"eefbb921df91cd4467ae6b9e5f89bbec09ca198ec38639dae6558d30d8e42a54"} Oct 31 07:33:03.593578 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:03.593547 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerStarted","Data":"35157f5ef5cef3696739a7b73586f3ebc34694b5ea494d676d881ec9ba34c8fc"} Oct 31 07:33:03.634409 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:03.634366 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" podStartSLOduration=3.906003378 podStartE2EDuration="16.634350086s" totalImagesPullingTime="12.728346708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:32:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:03.633623222 +0000 UTC m=+228.106017470" watchObservedRunningTime="2026-10-31 07:33:03.634350086 +0000 UTC m=+228.106744334" Oct 31 07:33:04.599004 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:04.598966 4870 generic.go:356] "Generic (PLEG): container finished" podID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerID="35157f5ef5cef3696739a7b73586f3ebc34694b5ea494d676d881ec9ba34c8fc" exitCode=0 Oct 31 07:33:04.599385 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:04.599054 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerDied","Data":"35157f5ef5cef3696739a7b73586f3ebc34694b5ea494d676d881ec9ba34c8fc"} Oct 31 07:33:05.728950 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.728921 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:33:05.776610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.776582 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util\") pod \"49c75a9c-90e1-4258-9c80-6ca14db2133c\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " Oct 31 07:33:05.776765 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.776625 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle\") pod \"49c75a9c-90e1-4258-9c80-6ca14db2133c\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " Oct 31 07:33:05.776765 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.776669 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p\" (UniqueName: \"kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p\") pod \"49c75a9c-90e1-4258-9c80-6ca14db2133c\" (UID: \"49c75a9c-90e1-4258-9c80-6ca14db2133c\") " Oct 31 07:33:05.777548 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.777513 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle" pod "49c75a9c-90e1-4258-9c80-6ca14db2133c" (UID: "49c75a9c-90e1-4258-9c80-6ca14db2133c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:33:05.778934 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.778910 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p" pod "49c75a9c-90e1-4258-9c80-6ca14db2133c" (UID: "49c75a9c-90e1-4258-9c80-6ca14db2133c"). InnerVolumeSpecName "kube-api-access-qnn7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:33:05.780924 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.780902 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util" pod "49c75a9c-90e1-4258-9c80-6ca14db2133c" (UID: "49c75a9c-90e1-4258-9c80-6ca14db2133c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:33:05.877424 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.877338 4870 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-util\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:33:05.877424 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.877367 4870 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/49c75a9c-90e1-4258-9c80-6ca14db2133c-bundle\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:33:05.877424 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:05.877379 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnn7p\" (UniqueName: \"kubernetes.io/projected/49c75a9c-90e1-4258-9c80-6ca14db2133c-kube-api-access-qnn7p\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:33:06.095930 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:06.095889 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:33:06.607502 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:06.607466 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" event={"ID":"49c75a9c-90e1-4258-9c80-6ca14db2133c","Type":"ContainerDied","Data":"d43bebe42cfa28232ca78afa84ed2607bcc1dbe8b72a00454e5daa005a9ac222"} Oct 31 07:33:06.607502 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:06.607501 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/98a21a31f7a8ae0212b9c22cc643f9202bfa37cdda9b658bc303ea820csnqs6" Oct 31 07:33:06.607712 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:06.607536 4870 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d43bebe42cfa28232ca78afa84ed2607bcc1dbe8b72a00454e5daa005a9ac222" Oct 31 07:33:06.620706 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:06.620684 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:09.508761 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:09.508725 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:09.627531 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:09.627500 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:10.900817 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.900749 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk"] Oct 31 07:33:10.901289 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901216 4870 cpu_manager.go:414] "containerMap: removing container" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="pull" Oct 31 07:33:10.901289 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901237 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="pull" Oct 31 07:33:10.901289 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901269 4870 cpu_manager.go:414] "containerMap: removing container" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="util" Oct 31 07:33:10.901289 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901276 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="util" Oct 31 07:33:10.901289 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901288 4870 cpu_manager.go:414] "containerMap: removing container" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="extract" Oct 31 07:33:10.901566 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901295 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="extract" Oct 31 07:33:10.901566 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.901387 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c75a9c-90e1-4258-9c80-6ca14db2133c" containerName="extract" Oct 31 07:33:10.942432 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.942398 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk"] Oct 31 07:33:10.942591 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.942445 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:10.957837 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.957807 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\"" Oct 31 07:33:10.957999 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.957983 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\"" Oct 31 07:33:10.958051 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.957988 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-kk7cc\"" Oct 31 07:33:10.958051 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:10.958010 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\"" Oct 31 07:33:11.013528 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.013503 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z946h\" (UniqueName: \"kubernetes.io/projected/33a8c492-e21d-41ba-ad0b-64cbbcf0fc19-kube-api-access-z946h\") pod \"custom-metrics-autoscaler-operator-c667576c8-8ncfk\" (UID: \"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.013631 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.013534 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/33a8c492-e21d-41ba-ad0b-64cbbcf0fc19-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-8ncfk\" (UID: \"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.114653 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.114614 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z946h\" (UniqueName: \"kubernetes.io/projected/33a8c492-e21d-41ba-ad0b-64cbbcf0fc19-kube-api-access-z946h\") pod \"custom-metrics-autoscaler-operator-c667576c8-8ncfk\" (UID: \"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.114653 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.114647 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/33a8c492-e21d-41ba-ad0b-64cbbcf0fc19-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-8ncfk\" (UID: \"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.116940 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.116916 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/33a8c492-e21d-41ba-ad0b-64cbbcf0fc19-certificates\") pod \"custom-metrics-autoscaler-operator-c667576c8-8ncfk\" (UID: \"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.137537 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.137510 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z946h\" (UniqueName: \"kubernetes.io/projected/33a8c492-e21d-41ba-ad0b-64cbbcf0fc19-kube-api-access-z946h\") pod \"custom-metrics-autoscaler-operator-c667576c8-8ncfk\" (UID: \"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19\") " pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.266625 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.266546 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:11.600531 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.600506 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk"] Oct 31 07:33:11.603512 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:33:11.603484 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a8c492_e21d_41ba_ad0b_64cbbcf0fc19.slice/crio-817b52b304c0e280b065303e331aaafde6fb90da4978edb89eacd5f102545a7b WatchSource:0}: Error finding container 817b52b304c0e280b065303e331aaafde6fb90da4978edb89eacd5f102545a7b: Status 404 returned error can't find the container with id 817b52b304c0e280b065303e331aaafde6fb90da4978edb89eacd5f102545a7b Oct 31 07:33:11.627588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:11.627559 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" event={"ID":"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19","Type":"ContainerStarted","Data":"817b52b304c0e280b065303e331aaafde6fb90da4978edb89eacd5f102545a7b"} Oct 31 07:33:15.644600 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:15.644565 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" event={"ID":"33a8c492-e21d-41ba-ad0b-64cbbcf0fc19","Type":"ContainerStarted","Data":"82ada567ce24428e2dfc68cf8b04ac00acd68b16b80656348a3e00b9f121b4a5"} Oct 31 07:33:15.645036 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:15.644686 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:15.743907 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:15.743858 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" podStartSLOduration=2.131819647 podStartE2EDuration="5.743843872s" totalImagesPullingTime="3.612024225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:15.73109915 +0000 UTC m=+240.203493397" watchObservedRunningTime="2026-10-31 07:33:15.743843872 +0000 UTC m=+240.216238111" Oct 31 07:33:15.800056 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:15.800021 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:16.394673 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.394633 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:16.405259 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.405230 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-7dc85965f-v8wjx"] Oct 31 07:33:16.421442 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.421419 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.427880 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.427857 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-v8wjx"] Oct 31 07:33:16.435403 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.435374 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\"" Oct 31 07:33:16.457369 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.457345 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4fb0acbe-fa33-4425-85ef-13e3214a32f1-cabundle0\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.457475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.457380 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.457475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.457400 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmsjc\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-kube-api-access-vmsjc\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.506204 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.506177 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\"" Oct 31 07:33:16.506332 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.506234 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-kd9hj\"" Oct 31 07:33:16.558568 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.558535 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4fb0acbe-fa33-4425-85ef-13e3214a32f1-cabundle0\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.558568 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.558572 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.558838 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.558592 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmsjc\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-kube-api-access-vmsjc\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.558838 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:16.558696 4870 secret.go:281] references non-existent secret key: ca.crt Oct 31 07:33:16.558838 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:16.558716 4870 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Oct 31 07:33:16.558838 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:16.558726 4870 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-v8wjx: references non-existent secret key: ca.crt Oct 31 07:33:16.558838 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:16.558774 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates podName:4fb0acbe-fa33-4425-85ef-13e3214a32f1 nodeName:}" failed. No retries permitted until 2026-10-31 07:33:17.0587596 +0000 UTC m=+241.531153843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates") pod "keda-operator-7dc85965f-v8wjx" (UID: "4fb0acbe-fa33-4425-85ef-13e3214a32f1") : references non-existent secret key: ca.crt Oct 31 07:33:16.559255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.559233 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4fb0acbe-fa33-4425-85ef-13e3214a32f1-cabundle0\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.583182 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.583152 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmsjc\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-kube-api-access-vmsjc\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:16.585010 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:16.584988 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:17.062057 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:17.062023 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:17.062447 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:17.062199 4870 secret.go:281] references non-existent secret key: ca.crt Oct 31 07:33:17.062447 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:17.062224 4870 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt Oct 31 07:33:17.062447 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:17.062239 4870 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-7dc85965f-v8wjx: references non-existent secret key: ca.crt Oct 31 07:33:17.062447 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:17.062322 4870 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates podName:4fb0acbe-fa33-4425-85ef-13e3214a32f1 nodeName:}" failed. No retries permitted until 2026-10-31 07:33:18.062295509 +0000 UTC m=+242.534689754 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates") pod "keda-operator-7dc85965f-v8wjx" (UID: "4fb0acbe-fa33-4425-85ef-13e3214a32f1") : references non-existent secret key: ca.crt Oct 31 07:33:17.160389 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:17.160358 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:17.263254 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:17.263222 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:17.460755 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:17.460673 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:17.609576 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:17.609544 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:17.717632 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:17.717511 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:18.069618 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.069579 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:18.072012 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.071991 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4fb0acbe-fa33-4425-85ef-13e3214a32f1-certificates\") pod \"keda-operator-7dc85965f-v8wjx\" (UID: \"4fb0acbe-fa33-4425-85ef-13e3214a32f1\") " pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:18.249879 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.249847 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:18.313338 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.313115 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:18.540998 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.540965 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-7dc85965f-v8wjx"] Oct 31 07:33:18.544141 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:33:18.544114 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fb0acbe_fa33_4425_85ef_13e3214a32f1.slice/crio-7788ea270f3cfe023c965c7aa56a9971ab15400e1d51fda84e58cbd048a4973c WatchSource:0}: Error finding container 7788ea270f3cfe023c965c7aa56a9971ab15400e1d51fda84e58cbd048a4973c: Status 404 returned error can't find the container with id 7788ea270f3cfe023c965c7aa56a9971ab15400e1d51fda84e58cbd048a4973c Oct 31 07:33:18.628300 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.628262 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:18.662072 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.662043 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" event={"ID":"4fb0acbe-fa33-4425-85ef-13e3214a32f1","Type":"ContainerStarted","Data":"7788ea270f3cfe023c965c7aa56a9971ab15400e1d51fda84e58cbd048a4973c"} Oct 31 07:33:18.894098 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.894015 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:18.995212 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:18.995183 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:21.675357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:21.675321 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" event={"ID":"4fb0acbe-fa33-4425-85ef-13e3214a32f1","Type":"ContainerStarted","Data":"bbda98313ccfa5c870d39fa652e5a73b095aeb5e96d4756e6e8d40ae951562a3"} Oct 31 07:33:21.675847 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:21.675459 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:21.714263 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:21.714213 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" podStartSLOduration=2.678854773 podStartE2EDuration="5.714201181s" totalImagesPullingTime="3.035346408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:33:21.713453594 +0000 UTC m=+246.185847839" watchObservedRunningTime="2026-10-31 07:33:21.714201181 +0000 UTC m=+246.186595491" Oct 31 07:33:30.199440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:30.199374 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:30.301206 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:30.301165 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:36.095624 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:33:36.095588 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:33:36.651268 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:36.651239 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-c667576c8-8ncfk" Oct 31 07:33:42.681808 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:42.681764 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-7dc85965f-v8wjx" Oct 31 07:33:43.324061 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:43.323988 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:33:44.515674 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:33:44.515632 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:06.095596 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:34:06.095555 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:34:06.741516 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:06.741481 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:09.944339 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:09.944306 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:10.049479 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:10.049450 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:15.675693 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:15.675667 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:34:15.677303 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:15.677281 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:34:36.096153 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:34:36.096075 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:34:43.208562 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.208524 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:43.300702 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.300669 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-nchtb"] Oct 31 07:34:43.303963 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.303940 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.320199 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.320178 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:43.329385 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.329362 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Oct 31 07:34:43.329385 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.329378 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Oct 31 07:34:43.329581 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.329435 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\"" Oct 31 07:34:43.329581 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.329459 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-pk649\"" Oct 31 07:34:43.330611 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.330591 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-nchtb"] Oct 31 07:34:43.337753 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.337734 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-86c86ffb6b-7whcp"] Oct 31 07:34:43.344252 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.344232 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.362251 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.362227 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-86c86ffb6b-7whcp"] Oct 31 07:34:43.407528 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.407497 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e327c0e-22a0-4302-b759-7a553343e059-cert\") pod \"llmisvc-controller-manager-86c86ffb6b-7whcp\" (UID: \"3e327c0e-22a0-4302-b759-7a553343e059\") " pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.407673 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.407556 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8r28\" (UniqueName: \"kubernetes.io/projected/3e327c0e-22a0-4302-b759-7a553343e059-kube-api-access-c8r28\") pod \"llmisvc-controller-manager-86c86ffb6b-7whcp\" (UID: \"3e327c0e-22a0-4302-b759-7a553343e059\") " pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.407743 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.407697 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5db4k\" (UniqueName: \"kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k\") pod \"kserve-controller-manager-76d8b77fbb-nchtb\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.407803 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.407750 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert\") pod \"kserve-controller-manager-76d8b77fbb-nchtb\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.418041 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.418018 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-qs9pb\"" Oct 31 07:34:43.418041 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.418018 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\"" Oct 31 07:34:43.425831 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.425782 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:43.508895 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.508843 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8r28\" (UniqueName: \"kubernetes.io/projected/3e327c0e-22a0-4302-b759-7a553343e059-kube-api-access-c8r28\") pod \"llmisvc-controller-manager-86c86ffb6b-7whcp\" (UID: \"3e327c0e-22a0-4302-b759-7a553343e059\") " pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.508895 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.508884 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5db4k\" (UniqueName: \"kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k\") pod \"kserve-controller-manager-76d8b77fbb-nchtb\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.509029 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.508907 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert\") pod \"kserve-controller-manager-76d8b77fbb-nchtb\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.509029 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.508947 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e327c0e-22a0-4302-b759-7a553343e059-cert\") pod \"llmisvc-controller-manager-86c86ffb6b-7whcp\" (UID: \"3e327c0e-22a0-4302-b759-7a553343e059\") " pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.511433 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.511414 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3e327c0e-22a0-4302-b759-7a553343e059-cert\") pod \"llmisvc-controller-manager-86c86ffb6b-7whcp\" (UID: \"3e327c0e-22a0-4302-b759-7a553343e059\") " pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.511520 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.511497 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert\") pod \"kserve-controller-manager-76d8b77fbb-nchtb\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.527357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.527334 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:43.530985 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.530966 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5db4k\" (UniqueName: \"kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k\") pod \"kserve-controller-manager-76d8b77fbb-nchtb\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.531108 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.531089 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8r28\" (UniqueName: \"kubernetes.io/projected/3e327c0e-22a0-4302-b759-7a553343e059-kube-api-access-c8r28\") pod \"llmisvc-controller-manager-86c86ffb6b-7whcp\" (UID: \"3e327c0e-22a0-4302-b759-7a553343e059\") " pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.616893 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.616864 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:43.656584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.656558 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:43.908834 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.908780 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-nchtb"] Oct 31 07:34:43.911598 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:34:43.911570 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb594032b_d089_443e_ad1e_b2ecadd69145.slice/crio-1ec2ee04d3767dc6c1b82b32caaf869242d09657cb7574736c663643b651c954 WatchSource:0}: Error finding container 1ec2ee04d3767dc6c1b82b32caaf869242d09657cb7574736c663643b651c954: Status 404 returned error can't find the container with id 1ec2ee04d3767dc6c1b82b32caaf869242d09657cb7574736c663643b651c954 Oct 31 07:34:43.912787 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.912766 4870 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 07:34:43.957051 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.957025 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-86c86ffb6b-7whcp"] Oct 31 07:34:43.959898 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:34:43.959866 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e327c0e_22a0_4302_b759_7a553343e059.slice/crio-f5464140c319b8758fc08c14f4566433d3c07fd50fa8ea673b41633c326dc37e WatchSource:0}: Error finding container f5464140c319b8758fc08c14f4566433d3c07fd50fa8ea673b41633c326dc37e: Status 404 returned error can't find the container with id f5464140c319b8758fc08c14f4566433d3c07fd50fa8ea673b41633c326dc37e Oct 31 07:34:43.996237 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.996204 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" event={"ID":"3e327c0e-22a0-4302-b759-7a553343e059","Type":"ContainerStarted","Data":"f5464140c319b8758fc08c14f4566433d3c07fd50fa8ea673b41633c326dc37e"} Oct 31 07:34:43.997344 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:43.997324 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" event={"ID":"b594032b-d089-443e-ad1e-b2ecadd69145","Type":"ContainerStarted","Data":"1ec2ee04d3767dc6c1b82b32caaf869242d09657cb7574736c663643b651c954"} Oct 31 07:34:44.723364 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:44.723329 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:44.826445 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:44.826407 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:44.928508 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:44.928473 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:34:47.013575 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:47.013541 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" event={"ID":"3e327c0e-22a0-4302-b759-7a553343e059","Type":"ContainerStarted","Data":"ceceef7ff4b8be44cb7d075dc3bc1d39d0a4318e070d7039c39ace1ecda8fa71"} Oct 31 07:34:47.014008 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:47.013610 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:34:47.015366 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:47.015346 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" event={"ID":"b594032b-d089-443e-ad1e-b2ecadd69145","Type":"ContainerStarted","Data":"d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a"} Oct 31 07:34:47.015488 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:47.015426 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:34:47.096355 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:47.096283 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" podStartSLOduration=1.118702543 podStartE2EDuration="4.096263874s" totalImagesPullingTime="2.977561331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:34:47.057063384 +0000 UTC m=+331.529457633" watchObservedRunningTime="2026-10-31 07:34:47.096263874 +0000 UTC m=+331.568658123" Oct 31 07:34:47.096620 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:34:47.096590 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" podStartSLOduration=1.149321004 podStartE2EDuration="4.096582231s" totalImagesPullingTime="2.947261227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:34:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:34:47.095084558 +0000 UTC m=+331.567478805" watchObservedRunningTime="2026-10-31 07:34:47.096582231 +0000 UTC m=+331.568976481" Oct 31 07:35:04.833063 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:04.833031 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:35:06.095344 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:35:06.095306 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:35:06.857784 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:06.857751 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:35:18.021903 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:18.021869 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-86c86ffb6b-7whcp" Oct 31 07:35:18.024866 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:18.024840 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:35:30.233051 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:30.233017 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:35:30.337966 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:30.337936 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:35:31.275224 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.275191 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-nchtb"] Oct 31 07:35:31.275855 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.275465 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" podUID="b594032b-d089-443e-ad1e-b2ecadd69145" containerName="manager" containerID="cri-o://d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a" gracePeriod=10 Oct 31 07:35:31.297315 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.297289 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-92njn"] Oct 31 07:35:31.299388 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.299370 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.331126 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.331091 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-92njn"] Oct 31 07:35:31.389542 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.389512 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3988a8e-de8b-4ae5-b244-0cb028b1ddad-cert\") pod \"kserve-controller-manager-76d8b77fbb-92njn\" (UID: \"b3988a8e-de8b-4ae5-b244-0cb028b1ddad\") " pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.389670 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.389649 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dbq9\" (UniqueName: \"kubernetes.io/projected/b3988a8e-de8b-4ae5-b244-0cb028b1ddad-kube-api-access-8dbq9\") pod \"kserve-controller-manager-76d8b77fbb-92njn\" (UID: \"b3988a8e-de8b-4ae5-b244-0cb028b1ddad\") " pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.490576 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.490539 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dbq9\" (UniqueName: \"kubernetes.io/projected/b3988a8e-de8b-4ae5-b244-0cb028b1ddad-kube-api-access-8dbq9\") pod \"kserve-controller-manager-76d8b77fbb-92njn\" (UID: \"b3988a8e-de8b-4ae5-b244-0cb028b1ddad\") " pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.490762 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.490594 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3988a8e-de8b-4ae5-b244-0cb028b1ddad-cert\") pod \"kserve-controller-manager-76d8b77fbb-92njn\" (UID: \"b3988a8e-de8b-4ae5-b244-0cb028b1ddad\") " pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.492919 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.492892 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3988a8e-de8b-4ae5-b244-0cb028b1ddad-cert\") pod \"kserve-controller-manager-76d8b77fbb-92njn\" (UID: \"b3988a8e-de8b-4ae5-b244-0cb028b1ddad\") " pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.512688 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.512666 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dbq9\" (UniqueName: \"kubernetes.io/projected/b3988a8e-de8b-4ae5-b244-0cb028b1ddad-kube-api-access-8dbq9\") pod \"kserve-controller-manager-76d8b77fbb-92njn\" (UID: \"b3988a8e-de8b-4ae5-b244-0cb028b1ddad\") " pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.523994 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.523974 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:35:31.591024 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.590986 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k\" (UniqueName: \"kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k\") pod \"b594032b-d089-443e-ad1e-b2ecadd69145\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " Oct 31 07:35:31.591183 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.591089 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert\" (UniqueName: \"kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert\") pod \"b594032b-d089-443e-ad1e-b2ecadd69145\" (UID: \"b594032b-d089-443e-ad1e-b2ecadd69145\") " Oct 31 07:35:31.593111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.593076 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k" pod "b594032b-d089-443e-ad1e-b2ecadd69145" (UID: "b594032b-d089-443e-ad1e-b2ecadd69145"). InnerVolumeSpecName "kube-api-access-5db4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:35:31.593227 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.593081 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert" pod "b594032b-d089-443e-ad1e-b2ecadd69145" (UID: "b594032b-d089-443e-ad1e-b2ecadd69145"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:35:31.652251 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.652215 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:31.692470 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.692438 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5db4k\" (UniqueName: \"kubernetes.io/projected/b594032b-d089-443e-ad1e-b2ecadd69145-kube-api-access-5db4k\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:35:31.692470 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.692470 4870 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b594032b-d089-443e-ad1e-b2ecadd69145-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:35:31.943956 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:31.943932 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-92njn"] Oct 31 07:35:31.946604 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:35:31.946574 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3988a8e_de8b_4ae5_b244_0cb028b1ddad.slice/crio-1a875d18995fb75bf615ea6f7bf7dd1702e6cc490f90a9add1bf2f55f24e9768 WatchSource:0}: Error finding container 1a875d18995fb75bf615ea6f7bf7dd1702e6cc490f90a9add1bf2f55f24e9768: Status 404 returned error can't find the container with id 1a875d18995fb75bf615ea6f7bf7dd1702e6cc490f90a9add1bf2f55f24e9768 Oct 31 07:35:32.192582 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.192542 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" event={"ID":"b3988a8e-de8b-4ae5-b244-0cb028b1ddad","Type":"ContainerStarted","Data":"1a875d18995fb75bf615ea6f7bf7dd1702e6cc490f90a9add1bf2f55f24e9768"} Oct 31 07:35:32.193763 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.193736 4870 generic.go:356] "Generic (PLEG): container finished" podID="b594032b-d089-443e-ad1e-b2ecadd69145" containerID="d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a" exitCode=0 Oct 31 07:35:32.193886 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.193786 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" Oct 31 07:35:32.193886 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.193841 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" event={"ID":"b594032b-d089-443e-ad1e-b2ecadd69145","Type":"ContainerDied","Data":"d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a"} Oct 31 07:35:32.194011 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.193901 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-nchtb" event={"ID":"b594032b-d089-443e-ad1e-b2ecadd69145","Type":"ContainerDied","Data":"1ec2ee04d3767dc6c1b82b32caaf869242d09657cb7574736c663643b651c954"} Oct 31 07:35:32.194011 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.193937 4870 scope.go:122] "RemoveContainer" containerID="d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a" Oct 31 07:35:32.201977 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.201961 4870 scope.go:122] "RemoveContainer" containerID="d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a" Oct 31 07:35:32.202236 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:35:32.202219 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a\": container with ID starting with d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a not found: ID does not exist" containerID="d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a" Oct 31 07:35:32.202282 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.202246 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a"} err="failed to get container status \"d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a\": rpc error: code = NotFound desc = could not find container \"d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a\": container with ID starting with d13218f1b4c9094c7e41e6fd2fad5a974ffdc8143e9c9146326ef64fa64f000a not found: ID does not exist" Oct 31 07:35:32.242070 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.242039 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-nchtb"] Oct 31 07:35:32.253426 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:32.253400 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-76d8b77fbb-nchtb"] Oct 31 07:35:33.199291 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:33.199252 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" event={"ID":"b3988a8e-de8b-4ae5-b244-0cb028b1ddad","Type":"ContainerStarted","Data":"a88e28f552c7ed9a95a95ced087fc044e7ffba81b942dc4b16a25bc44318113d"} Oct 31 07:35:33.199764 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:33.199327 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:35:33.238311 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:33.238263 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" podStartSLOduration=1.861309613 podStartE2EDuration="2.238249757s" totalImagesPullingTime="376.940144ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:35:33.237151447 +0000 UTC m=+377.709545695" watchObservedRunningTime="2026-10-31 07:35:33.238249757 +0000 UTC m=+377.710644042" Oct 31 07:35:33.661907 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:33.661874 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b594032b-d089-443e-ad1e-b2ecadd69145" path="/var/lib/kubelet/pods/b594032b-d089-443e-ad1e-b2ecadd69145/volumes" Oct 31 07:35:36.095598 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:35:36.095559 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:35:59.617618 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:35:59.617535 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:01.592390 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:01.592359 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:04.210224 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:04.210192 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-76d8b77fbb-92njn" Oct 31 07:36:06.095720 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:36:06.095679 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:36:06.193263 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:06.193228 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:06.308557 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:06.308521 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:06.424597 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:06.424515 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:06.534323 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:06.534293 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:06.987375 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:06.987341 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:10.173900 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:10.173864 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:10.284476 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:36:10.284443 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:36:36.096229 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:36:36.096190 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:37:06.095944 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:37:06.095904 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:37:07.110343 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:07.110307 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:36.096581 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:37:36.096508 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:37:41.447664 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:41.447627 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:41.555975 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:41.555926 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:50.589702 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:50.589663 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:50.703041 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:50.703009 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:57.080119 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:57.080082 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:57.182509 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:57.182478 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:58.583952 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:58.583920 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:58.848667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:58.848594 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:58.963006 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:58.962969 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:59.134197 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.134123 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr"] Oct 31 07:37:59.134450 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.134438 4870 cpu_manager.go:414] "containerMap: removing container" podUID="b594032b-d089-443e-ad1e-b2ecadd69145" containerName="manager" Oct 31 07:37:59.134513 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.134452 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b594032b-d089-443e-ad1e-b2ecadd69145" containerName="manager" Oct 31 07:37:59.134513 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.134509 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="b594032b-d089-443e-ad1e-b2ecadd69145" containerName="manager" Oct 31 07:37:59.137470 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.137450 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.162929 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162897 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:37:59.162929 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162909 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:37:59.162929 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162909 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Oct 31 07:37:59.163132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162912 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:37:59.163132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162942 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:37:59.163132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162916 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:37:59.163132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.162966 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-rollback-predictor-serving-cert\"" Oct 31 07:37:59.164350 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.164333 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr"] Oct 31 07:37:59.290938 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.290907 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:59.321023 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.320995 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.321183 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.321030 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.321183 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.321058 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.321183 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.321131 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.321297 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.321227 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbzn\" (UniqueName: \"kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.396381 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.396305 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:37:59.422511 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.422475 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.422672 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.422517 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.422672 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.422548 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.422672 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.422566 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.422672 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.422604 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbzn\" (UniqueName: \"kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.422976 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.422956 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.423274 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.423250 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.423360 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.423296 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.424991 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.424975 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.444391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.444366 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbzn\" (UniqueName: \"kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn\") pod \"isvc-canary-rollback-predictor-5b7b487756-tlgzr\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.454261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.454237 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:37:59.760083 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.760055 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr"] Oct 31 07:37:59.762768 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:37:59.762737 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5241cf41_a060_407c_887e_7767b23e5347.slice/crio-7980c6b95ff85eae1082e6e366bfaf66ff9b0dc6c0e9a01d6d75b00a8e2509ab WatchSource:0}: Error finding container 7980c6b95ff85eae1082e6e366bfaf66ff9b0dc6c0e9a01d6d75b00a8e2509ab: Status 404 returned error can't find the container with id 7980c6b95ff85eae1082e6e366bfaf66ff9b0dc6c0e9a01d6d75b00a8e2509ab Oct 31 07:37:59.780616 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:37:59.780592 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerStarted","Data":"7980c6b95ff85eae1082e6e366bfaf66ff9b0dc6c0e9a01d6d75b00a8e2509ab"} Oct 31 07:38:03.803343 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:03.803307 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerStarted","Data":"29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879"} Oct 31 07:38:06.096465 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:38:06.096428 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:38:07.228266 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:07.228190 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:38:07.820328 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:07.820284 4870 generic.go:356] "Generic (PLEG): container finished" podID="5241cf41-a060-407c-887e-7767b23e5347" containerID="29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879" exitCode=0 Oct 31 07:38:07.820509 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:07.820357 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerDied","Data":"29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879"} Oct 31 07:38:19.382016 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:19.381950 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:38:19.484442 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:19.484401 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:38:21.892045 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:21.892004 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerStarted","Data":"83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897"} Oct 31 07:38:23.902910 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:23.902879 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerStarted","Data":"af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a"} Oct 31 07:38:23.903297 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:23.903027 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:38:23.958230 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:23.958142 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podStartSLOduration=2.002935006 podStartE2EDuration="24.95812914s" totalImagesPullingTime="19.035349749s" totalInitContainerRuntime="3.919844385s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:37:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:38:23.956723919 +0000 UTC m=+548.429118177" watchObservedRunningTime="2026-10-31 07:38:23.95812914 +0000 UTC m=+548.430523387" Oct 31 07:38:24.907138 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:24.907103 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:38:24.908619 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:24.908588 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:38:25.910125 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:25.910082 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:38:30.914777 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:30.914747 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:38:30.915261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:30.915234 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:38:36.096281 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:38:36.096171 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:38:40.915570 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:40.915529 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:38:50.915584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:38:50.915543 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:39:00.916158 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:00.916079 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:39:06.096312 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:39:06.096273 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:39:07.352527 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:07.352490 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:39:10.915737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:10.915689 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:39:15.703106 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:15.703077 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:39:15.710402 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:15.710374 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:39:20.915850 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:20.915809 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:39:30.915920 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:30.915884 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:39:33.657217 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:33.657178 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:39:36.096254 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:39:36.096219 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:39:43.661055 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:39:43.661019 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:40:06.096489 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:40:06.096448 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:40:07.461807 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:07.461755 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:40:09.028201 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:09.028169 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:40:14.535149 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:14.535116 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:40:36.096014 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:40:36.095936 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:40:46.429221 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:46.429185 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:40:46.545125 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:46.545089 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:40:46.830111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:46.830083 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:40:46.989274 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:40:46.989243 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:06.096636 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:41:06.096591 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:41:07.586669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:07.586635 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:36.095457 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:41:36.095421 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:41:49.359781 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:49.359724 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr"] Oct 31 07:41:49.360405 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:49.360086 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" containerID="cri-o://83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897" gracePeriod=30 Oct 31 07:41:49.360405 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:49.360097 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kube-rbac-proxy" containerID="cri-o://af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a" gracePeriod=30 Oct 31 07:41:49.412202 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:49.412169 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:49.734132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:49.734041 4870 generic.go:356] "Generic (PLEG): container finished" podID="5241cf41-a060-407c-887e-7767b23e5347" containerID="af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a" exitCode=0 Oct 31 07:41:49.734132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:49.734111 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerDied","Data":"af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a"} Oct 31 07:41:50.911117 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:50.911078 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused" Oct 31 07:41:53.657305 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:53.657253 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused" Oct 31 07:41:54.203377 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.203352 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:41:54.293381 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293357 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls\") pod \"5241cf41-a060-407c-887e-7767b23e5347\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " Oct 31 07:41:54.293533 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293402 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert\") pod \"5241cf41-a060-407c-887e-7767b23e5347\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " Oct 31 07:41:54.293533 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293470 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location\") pod \"5241cf41-a060-407c-887e-7767b23e5347\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " Oct 31 07:41:54.293533 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293486 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config\") pod \"5241cf41-a060-407c-887e-7767b23e5347\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " Oct 31 07:41:54.293533 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293513 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn\" (UniqueName: \"kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn\") pod \"5241cf41-a060-407c-887e-7767b23e5347\" (UID: \"5241cf41-a060-407c-887e-7767b23e5347\") " Oct 31 07:41:54.293878 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293843 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location" pod "5241cf41-a060-407c-887e-7767b23e5347" (UID: "5241cf41-a060-407c-887e-7767b23e5347"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:41:54.293995 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293899 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert" pod "5241cf41-a060-407c-887e-7767b23e5347" (UID: "5241cf41-a060-407c-887e-7767b23e5347"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:41:54.293995 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.293904 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config" pod "5241cf41-a060-407c-887e-7767b23e5347" (UID: "5241cf41-a060-407c-887e-7767b23e5347"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:41:54.295642 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.295620 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls" pod "5241cf41-a060-407c-887e-7767b23e5347" (UID: "5241cf41-a060-407c-887e-7767b23e5347"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:41:54.295728 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.295703 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn" pod "5241cf41-a060-407c-887e-7767b23e5347" (UID: "5241cf41-a060-407c-887e-7767b23e5347"). InnerVolumeSpecName "kube-api-access-lxbzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:41:54.394987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.394945 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5241cf41-a060-407c-887e-7767b23e5347-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:41:54.394987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.394984 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:41:54.395163 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.395000 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbzn\" (UniqueName: \"kubernetes.io/projected/5241cf41-a060-407c-887e-7767b23e5347-kube-api-access-lxbzn\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:41:54.395163 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.395013 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5241cf41-a060-407c-887e-7767b23e5347-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:41:54.395163 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.395026 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5241cf41-a060-407c-887e-7767b23e5347-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:41:54.756872 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.756774 4870 generic.go:356] "Generic (PLEG): container finished" podID="5241cf41-a060-407c-887e-7767b23e5347" containerID="83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897" exitCode=0 Oct 31 07:41:54.756872 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.756845 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" Oct 31 07:41:54.756872 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.756854 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerDied","Data":"83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897"} Oct 31 07:41:54.757462 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.756918 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr" event={"ID":"5241cf41-a060-407c-887e-7767b23e5347","Type":"ContainerDied","Data":"7980c6b95ff85eae1082e6e366bfaf66ff9b0dc6c0e9a01d6d75b00a8e2509ab"} Oct 31 07:41:54.757462 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.756945 4870 scope.go:122] "RemoveContainer" containerID="af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a" Oct 31 07:41:54.766080 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.766061 4870 scope.go:122] "RemoveContainer" containerID="83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897" Oct 31 07:41:54.773640 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.773618 4870 scope.go:122] "RemoveContainer" containerID="29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879" Oct 31 07:41:54.780852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.780835 4870 scope.go:122] "RemoveContainer" containerID="af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a" Oct 31 07:41:54.781126 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:41:54.781108 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a\": container with ID starting with af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a not found: ID does not exist" containerID="af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a" Oct 31 07:41:54.781179 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.781135 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a"} err="failed to get container status \"af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a\": rpc error: code = NotFound desc = could not find container \"af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a\": container with ID starting with af22a85809de65baf8443522aa6e1a80aaab39c3681b6503151ee29697d0400a not found: ID does not exist" Oct 31 07:41:54.781179 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.781155 4870 scope.go:122] "RemoveContainer" containerID="83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897" Oct 31 07:41:54.781376 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:41:54.781357 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897\": container with ID starting with 83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897 not found: ID does not exist" containerID="83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897" Oct 31 07:41:54.781414 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.781382 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897"} err="failed to get container status \"83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897\": rpc error: code = NotFound desc = could not find container \"83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897\": container with ID starting with 83f8fe5fa5d91361673b00e68d9808e6523f9a79efd34a313d7bc79df67b3897 not found: ID does not exist" Oct 31 07:41:54.781414 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.781398 4870 scope.go:122] "RemoveContainer" containerID="29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879" Oct 31 07:41:54.781576 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:41:54.781563 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879\": container with ID starting with 29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879 not found: ID does not exist" containerID="29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879" Oct 31 07:41:54.781616 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.781580 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879"} err="failed to get container status \"29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879\": rpc error: code = NotFound desc = could not find container \"29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879\": container with ID starting with 29a5adfd7e7082b5988cb7236ed137eb8454ccba598bc9df27cb26b58e6e0879 not found: ID does not exist" Oct 31 07:41:54.818479 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.818453 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr"] Oct 31 07:41:54.830094 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:54.830075 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-rollback-predictor-5b7b487756-tlgzr"] Oct 31 07:41:55.661104 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:55.661071 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5241cf41-a060-407c-887e-7767b23e5347" path="/var/lib/kubelet/pods/5241cf41-a060-407c-887e-7767b23e5347/volumes" Oct 31 07:41:56.977977 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:56.977946 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:57.098462 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098431 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5"] Oct 31 07:41:57.098852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098834 4870 cpu_manager.go:414] "containerMap: removing container" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kube-rbac-proxy" Oct 31 07:41:57.098986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098853 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kube-rbac-proxy" Oct 31 07:41:57.098986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098893 4870 cpu_manager.go:414] "containerMap: removing container" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" Oct 31 07:41:57.098986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098900 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" Oct 31 07:41:57.098986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098914 4870 cpu_manager.go:414] "containerMap: removing container" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="storage-initializer" Oct 31 07:41:57.098986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.098921 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="storage-initializer" Oct 31 07:41:57.099244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.099004 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kube-rbac-proxy" Oct 31 07:41:57.099244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.099022 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="5241cf41-a060-407c-887e-7767b23e5347" containerName="kserve-container" Oct 31 07:41:57.103591 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.103568 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.123482 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.123461 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:41:57.134196 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.134172 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:57.135164 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.135144 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:41:57.135450 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.135427 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-stop-v2-predictor-serving-cert\"" Oct 31 07:41:57.135563 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.135450 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Oct 31 07:41:57.135563 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.135436 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:41:57.135563 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.135460 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:41:57.135563 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.135475 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:41:57.137665 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.137639 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5"] Oct 31 07:41:57.217838 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.217750 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.217989 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.217837 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.217989 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.217930 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.217989 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.217960 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fzt6\" (UniqueName: \"kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.218122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.217986 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.248820 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.248737 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:57.318899 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.318867 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.318899 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.318900 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fzt6\" (UniqueName: \"kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.319078 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.318916 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.319078 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.319030 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.319158 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.319089 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.319504 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.319480 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.319640 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.319618 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.319699 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.319623 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.321569 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.321546 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.341253 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.341230 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fzt6\" (UniqueName: \"kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6\") pod \"isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.403370 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.403333 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d"] Oct 31 07:41:57.407276 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.407259 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.412178 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.412146 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:41:57.420314 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.420295 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:41:57.421711 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.421693 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-stop-predictor-serving-cert\"" Oct 31 07:41:57.430917 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.430893 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d"] Oct 31 07:41:57.521361 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.521289 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.521361 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.521331 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.521569 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.521445 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.521569 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.521491 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.521725 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.521579 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjb95\" (UniqueName: \"kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.622900 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.622865 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623062 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.622919 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623062 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.622950 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623062 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.622974 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjb95\" (UniqueName: \"kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623062 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.622995 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623368 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.623341 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623616 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.623598 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.623672 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.623631 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.625539 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.625513 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.645314 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.645286 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjb95\" (UniqueName: \"kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95\") pod \"isvc-canary-stop-predictor-589d977cbd-7nq4d\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.724872 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.724842 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:41:57.729207 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.729097 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5"] Oct 31 07:41:57.731490 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:41:57.731439 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77b29c0d_f9ea_4a53_8b7b_f57ada1f9040.slice/crio-fa47e3b3248ebac15501141b542e2e12d73acfbb967f7f61cde98585b9e8112c WatchSource:0}: Error finding container fa47e3b3248ebac15501141b542e2e12d73acfbb967f7f61cde98585b9e8112c: Status 404 returned error can't find the container with id fa47e3b3248ebac15501141b542e2e12d73acfbb967f7f61cde98585b9e8112c Oct 31 07:41:57.773732 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:57.773666 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerStarted","Data":"fa47e3b3248ebac15501141b542e2e12d73acfbb967f7f61cde98585b9e8112c"} Oct 31 07:41:58.034270 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:58.034244 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d"] Oct 31 07:41:58.037204 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:41:58.037174 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf96dd0_e2f1_4b66_a4b5_d3f81b7983a1.slice/crio-0545e2a9d789a27b11d0bb69bc7b6ab694824b0cb529124521731b3962aa43aa WatchSource:0}: Error finding container 0545e2a9d789a27b11d0bb69bc7b6ab694824b0cb529124521731b3962aa43aa: Status 404 returned error can't find the container with id 0545e2a9d789a27b11d0bb69bc7b6ab694824b0cb529124521731b3962aa43aa Oct 31 07:41:58.780256 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:58.780215 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerStarted","Data":"a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc"} Oct 31 07:41:58.781852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:58.781821 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerStarted","Data":"ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75"} Oct 31 07:41:58.781985 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:41:58.781881 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerStarted","Data":"0545e2a9d789a27b11d0bb69bc7b6ab694824b0cb529124521731b3962aa43aa"} Oct 31 07:42:02.800298 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:02.800265 4870 generic.go:356] "Generic (PLEG): container finished" podID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerID="ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75" exitCode=0 Oct 31 07:42:02.800755 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:02.800348 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerDied","Data":"ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75"} Oct 31 07:42:02.801893 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:02.801867 4870 generic.go:356] "Generic (PLEG): container finished" podID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerID="a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc" exitCode=0 Oct 31 07:42:02.802007 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:02.801898 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerDied","Data":"a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc"} Oct 31 07:42:03.808041 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.808006 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerStarted","Data":"fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0"} Oct 31 07:42:03.808555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.808081 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerStarted","Data":"1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28"} Oct 31 07:42:03.808555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.808396 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:42:03.808555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.808451 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:42:03.810113 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.810084 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerStarted","Data":"703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e"} Oct 31 07:42:03.810226 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.810142 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerStarted","Data":"134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e"} Oct 31 07:42:03.810226 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.810151 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:03.810386 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.810367 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:42:03.810439 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.810419 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:42:03.811505 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.811480 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:03.875255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.875213 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podStartSLOduration=2.97801033 podStartE2EDuration="6.875200487s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.897190157s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:03.862429429 +0000 UTC m=+768.334823688" watchObservedRunningTime="2026-10-31 07:42:03.875200487 +0000 UTC m=+768.347594735" Oct 31 07:42:03.904841 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:03.904780 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podStartSLOduration=2.678989391 podStartE2EDuration="6.904767691s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.2257783s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:41:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:42:03.903959689 +0000 UTC m=+768.376353950" watchObservedRunningTime="2026-10-31 07:42:03.904767691 +0000 UTC m=+768.377161938" Oct 31 07:42:04.815045 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:04.815007 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:04.815489 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:04.815008 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:06.095554 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:42:06.095517 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:42:07.707521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:07.707488 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:09.819526 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:09.819494 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:42:09.820019 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:09.819597 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:42:09.820091 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:09.820025 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:09.820091 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:09.820068 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:19.820500 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:19.820461 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:19.820500 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:19.820491 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:29.820624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:29.820583 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:29.821124 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:29.820583 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:36.096627 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:42:36.096585 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:42:36.711151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:36.711120 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:36.845757 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:36.845721 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:39.820010 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:39.819974 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:39.820390 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:39.820040 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:47.274646 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:47.274614 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:47.378180 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:47.378149 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:49.820852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:49.820815 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:49.821270 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:49.820820 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:42:54.768147 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:54.768113 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:54.875309 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:54.875275 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:55.425424 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:55.425393 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:55.586415 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:55.586385 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:42:59.820952 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:59.820913 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:42:59.821420 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:42:59.820952 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:43:06.095861 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:06.095825 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:43:07.828888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:07.828856 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:43:09.820050 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:09.820009 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:43:09.820613 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:09.820226 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:43:19.820309 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:19.820274 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused" Oct 31 07:43:19.820786 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:19.820282 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:43:22.657545 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:22.657507 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:43:29.820705 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:29.820676 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:43:30.307194 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:30.307158 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d"] Oct 31 07:43:30.307497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:30.307425 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" containerID="cri-o://1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28" gracePeriod=30 Oct 31 07:43:30.307682 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:30.307486 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kube-rbac-proxy" containerID="cri-o://fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0" gracePeriod=30 Oct 31 07:43:31.175754 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:31.175719 4870 generic.go:356] "Generic (PLEG): container finished" podID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerID="fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0" exitCode=0 Oct 31 07:43:31.176157 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:31.175809 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerDied","Data":"fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0"} Oct 31 07:43:32.135414 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:32.135383 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5"] Oct 31 07:43:32.135639 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:32.135614 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" containerID="cri-o://134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e" gracePeriod=30 Oct 31 07:43:32.135722 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:32.135684 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kube-rbac-proxy" containerID="cri-o://703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e" gracePeriod=30 Oct 31 07:43:32.145389 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:32.145364 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:43:32.312016 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:32.311985 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:43:32.657589 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:32.657553 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused" Oct 31 07:43:33.186234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:33.186199 4870 generic.go:356] "Generic (PLEG): container finished" podID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerID="703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e" exitCode=0 Oct 31 07:43:33.186404 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:33.186268 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerDied","Data":"703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e"} Oct 31 07:43:34.816016 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:34.815971 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused" Oct 31 07:43:34.816406 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:34.815971 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused" Oct 31 07:43:35.346138 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.346115 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:43:35.511177 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511105 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert\") pod \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " Oct 31 07:43:35.511177 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511153 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config\") pod \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " Oct 31 07:43:35.511351 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511182 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location\") pod \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " Oct 31 07:43:35.511391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511345 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95\" (UniqueName: \"kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95\") pod \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " Oct 31 07:43:35.511438 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511396 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls\") pod \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\" (UID: \"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1\") " Oct 31 07:43:35.511501 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511475 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location" pod "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" (UID: "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:43:35.511556 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511502 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert" pod "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" (UID: "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:43:35.511556 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511514 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config" pod "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" (UID: "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:43:35.511731 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511705 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:35.511787 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511737 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:35.511787 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.511755 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:35.513444 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.513420 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95" pod "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" (UID: "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1"). InnerVolumeSpecName "kube-api-access-vjb95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:43:35.513547 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.513526 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls" pod "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" (UID: "aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:43:35.612219 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.612181 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjb95\" (UniqueName: \"kubernetes.io/projected/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-kube-api-access-vjb95\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:35.612219 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:35.612215 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:36.095152 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:36.095112 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:43:36.200777 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.200667 4870 generic.go:356] "Generic (PLEG): container finished" podID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerID="1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28" exitCode=0 Oct 31 07:43:36.200777 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.200729 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerDied","Data":"1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28"} Oct 31 07:43:36.200777 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.200742 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" Oct 31 07:43:36.200777 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.200780 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d" event={"ID":"aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1","Type":"ContainerDied","Data":"0545e2a9d789a27b11d0bb69bc7b6ab694824b0cb529124521731b3962aa43aa"} Oct 31 07:43:36.214763 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.200829 4870 scope.go:122] "RemoveContainer" containerID="fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0" Oct 31 07:43:36.214763 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.208860 4870 scope.go:122] "RemoveContainer" containerID="1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28" Oct 31 07:43:36.216106 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.216087 4870 scope.go:122] "RemoveContainer" containerID="ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75" Oct 31 07:43:36.222872 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.222854 4870 scope.go:122] "RemoveContainer" containerID="fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0" Oct 31 07:43:36.223118 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:36.223096 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0\": container with ID starting with fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0 not found: ID does not exist" containerID="fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0" Oct 31 07:43:36.223198 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.223129 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0"} err="failed to get container status \"fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0\": rpc error: code = NotFound desc = could not find container \"fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0\": container with ID starting with fa187d41fc143670bff8a8ab4f02c4295c7806214c499c1e2bbc00a7859597f0 not found: ID does not exist" Oct 31 07:43:36.223198 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.223157 4870 scope.go:122] "RemoveContainer" containerID="1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28" Oct 31 07:43:36.223402 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:36.223383 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28\": container with ID starting with 1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28 not found: ID does not exist" containerID="1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28" Oct 31 07:43:36.223449 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.223409 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28"} err="failed to get container status \"1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28\": rpc error: code = NotFound desc = could not find container \"1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28\": container with ID starting with 1c6cdd29dc509386d449de9151c8a7ea429c50325c95bbbe76b13aa382ae9d28 not found: ID does not exist" Oct 31 07:43:36.223449 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.223427 4870 scope.go:122] "RemoveContainer" containerID="ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75" Oct 31 07:43:36.223625 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:36.223609 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75\": container with ID starting with ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75 not found: ID does not exist" containerID="ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75" Oct 31 07:43:36.223669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.223627 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75"} err="failed to get container status \"ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75\": rpc error: code = NotFound desc = could not find container \"ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75\": container with ID starting with ceead7445d0adf3def1d1bcb21bccb64f7da43b6a1e3810be1ccfcefbda65c75 not found: ID does not exist" Oct 31 07:43:36.262858 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.262828 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d"] Oct 31 07:43:36.274752 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:36.274718 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-predictor-589d977cbd-7nq4d"] Oct 31 07:43:37.184434 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.184412 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:43:37.208168 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.208143 4870 generic.go:356] "Generic (PLEG): container finished" podID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerID="134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e" exitCode=0 Oct 31 07:43:37.208285 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.208181 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerDied","Data":"134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e"} Oct 31 07:43:37.208285 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.208197 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" Oct 31 07:43:37.208285 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.208226 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5" event={"ID":"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040","Type":"ContainerDied","Data":"fa47e3b3248ebac15501141b542e2e12d73acfbb967f7f61cde98585b9e8112c"} Oct 31 07:43:37.208285 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.208252 4870 scope.go:122] "RemoveContainer" containerID="703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e" Oct 31 07:43:37.215903 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.215876 4870 scope.go:122] "RemoveContainer" containerID="134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e" Oct 31 07:43:37.222973 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.222951 4870 scope.go:122] "RemoveContainer" containerID="a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc" Oct 31 07:43:37.229743 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.229722 4870 scope.go:122] "RemoveContainer" containerID="703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e" Oct 31 07:43:37.230015 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:37.229996 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e\": container with ID starting with 703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e not found: ID does not exist" containerID="703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e" Oct 31 07:43:37.230089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.230025 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e"} err="failed to get container status \"703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e\": rpc error: code = NotFound desc = could not find container \"703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e\": container with ID starting with 703290b0cdcb526bcc6c24bb917e0aff2028285256d46b269cfefa073407c88e not found: ID does not exist" Oct 31 07:43:37.230089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.230047 4870 scope.go:122] "RemoveContainer" containerID="134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e" Oct 31 07:43:37.230296 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:37.230278 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e\": container with ID starting with 134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e not found: ID does not exist" containerID="134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e" Oct 31 07:43:37.230334 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.230303 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e"} err="failed to get container status \"134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e\": rpc error: code = NotFound desc = could not find container \"134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e\": container with ID starting with 134fddc6748a3caf9cc97a4ed56b07b5c261c99a78688ce7bb2c8fd39f15fc4e not found: ID does not exist" Oct 31 07:43:37.230334 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.230318 4870 scope.go:122] "RemoveContainer" containerID="a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc" Oct 31 07:43:37.230547 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:43:37.230529 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc\": container with ID starting with a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc not found: ID does not exist" containerID="a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc" Oct 31 07:43:37.230599 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.230555 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc"} err="failed to get container status \"a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc\": rpc error: code = NotFound desc = could not find container \"a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc\": container with ID starting with a4f167bdd06d3f42f465484fb4d1d2ae69bde2c00058f6a69e1ce1802ba783dc not found: ID does not exist" Oct 31 07:43:37.326272 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326237 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location\") pod \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " Oct 31 07:43:37.326458 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326289 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert\") pod \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " Oct 31 07:43:37.326458 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326315 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls\") pod \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " Oct 31 07:43:37.326458 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326342 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6\" (UniqueName: \"kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6\") pod \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " Oct 31 07:43:37.326458 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326380 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config\") pod \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\" (UID: \"77b29c0d-f9ea-4a53-8b7b-f57ada1f9040\") " Oct 31 07:43:37.326734 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326703 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location" pod "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" (UID: "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:43:37.326856 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326741 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert" pod "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" (UID: "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:43:37.326856 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.326816 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config" pod "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" (UID: "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:43:37.328465 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.328444 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6" pod "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" (UID: "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040"). InnerVolumeSpecName "kube-api-access-8fzt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:43:37.328519 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.328475 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls" pod "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" (UID: "77b29c0d-f9ea-4a53-8b7b-f57ada1f9040"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:43:37.427936 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.427905 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:37.427936 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.427930 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:37.427936 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.427940 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:37.428146 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.427950 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fzt6\" (UniqueName: \"kubernetes.io/projected/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-api-access-8fzt6\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:37.428146 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.427959 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:43:37.551326 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.551293 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5"] Oct 31 07:43:37.563268 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.563234 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-stop-v2-predictor-76f9b6b879-dn9d5"] Oct 31 07:43:37.661197 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.661167 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" path="/var/lib/kubelet/pods/77b29c0d-f9ea-4a53-8b7b-f57ada1f9040/volumes" Oct 31 07:43:37.661663 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:37.661651 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" path="/var/lib/kubelet/pods/aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1/volumes" Oct 31 07:43:38.862075 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.862046 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:43:38.989309 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989265 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg"] Oct 31 07:43:38.989639 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989615 4870 cpu_manager.go:414] "containerMap: removing container" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kube-rbac-proxy" Oct 31 07:43:38.989639 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989631 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kube-rbac-proxy" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989655 4870 cpu_manager.go:414] "containerMap: removing container" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="storage-initializer" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989663 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="storage-initializer" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989673 4870 cpu_manager.go:414] "containerMap: removing container" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="storage-initializer" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989681 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="storage-initializer" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989700 4870 cpu_manager.go:414] "containerMap: removing container" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989709 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989732 4870 cpu_manager.go:414] "containerMap: removing container" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kube-rbac-proxy" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989739 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kube-rbac-proxy" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989766 4870 cpu_manager.go:414] "containerMap: removing container" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" Oct 31 07:43:38.989846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989773 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" Oct 31 07:43:38.990159 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989900 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kube-rbac-proxy" Oct 31 07:43:38.990159 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989912 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kserve-container" Oct 31 07:43:38.990159 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989918 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="77b29c0d-f9ea-4a53-8b7b-f57ada1f9040" containerName="kube-rbac-proxy" Oct 31 07:43:38.990159 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.989927 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf96dd0-e2f1-4b66-a4b5-d3f81b7983a1" containerName="kserve-container" Oct 31 07:43:38.994488 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.994468 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:38.996135 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:38.996115 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:43:39.009455 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.009430 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:43:39.009555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.009488 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-primary-d46a81-predictor-serving-cert\"" Oct 31 07:43:39.009555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.009532 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:43:39.021151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.021081 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:43:39.021151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.021093 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:43:39.021151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.021119 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-primary-d46a81-kube-rbac-proxy-sar-config\"" Oct 31 07:43:39.021151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.021121 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:43:39.022929 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.022911 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg"] Oct 31 07:43:39.143027 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.142960 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczpn\" (UniqueName: \"kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.143027 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.142997 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.143225 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.143091 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.143225 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.143206 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.143309 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.143246 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244099 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244132 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244135 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244321 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244169 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczpn\" (UniqueName: \"kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244321 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244196 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244321 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244242 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244527 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244998 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244964 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.244998 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.244965 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.246726 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.246705 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.266706 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.266684 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczpn\" (UniqueName: \"kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn\") pod \"isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.311612 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.311574 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:39.619262 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:39.619227 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg"] Oct 31 07:43:39.622339 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:43:39.622308 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ed22fe2_e63a_4de8_a2c7_79c5cd366638.slice/crio-09eef8ffafa164df75ee5191347eed154ae2728d668551dc182414c7f41fec96 WatchSource:0}: Error finding container 09eef8ffafa164df75ee5191347eed154ae2728d668551dc182414c7f41fec96: Status 404 returned error can't find the container with id 09eef8ffafa164df75ee5191347eed154ae2728d668551dc182414c7f41fec96 Oct 31 07:43:40.230290 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:40.230252 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerStarted","Data":"36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13"} Oct 31 07:43:40.230682 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:40.230333 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerStarted","Data":"09eef8ffafa164df75ee5191347eed154ae2728d668551dc182414c7f41fec96"} Oct 31 07:43:44.248039 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:44.248006 4870 generic.go:356] "Generic (PLEG): container finished" podID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerID="36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13" exitCode=0 Oct 31 07:43:44.248450 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:44.248077 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerDied","Data":"36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13"} Oct 31 07:43:45.255221 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:45.255185 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerStarted","Data":"3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443"} Oct 31 07:43:45.255591 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:45.255249 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerStarted","Data":"85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1"} Oct 31 07:43:45.255591 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:45.255427 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:45.311838 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:45.311767 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podStartSLOduration=3.478767058 podStartE2EDuration="7.311750718s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.83298366s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:43:45.309772181 +0000 UTC m=+869.782166429" watchObservedRunningTime="2026-10-31 07:43:45.311750718 +0000 UTC m=+869.784144968" Oct 31 07:43:46.260537 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:46.260501 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:46.262156 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:46.262119 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:43:47.264990 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:47.264953 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:43:52.269988 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:52.269960 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:43:52.270616 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:43:52.270588 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:02.271496 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:02.271456 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:06.096706 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:44:06.096667 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:44:07.945638 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:07.945603 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:44:12.271105 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:12.271066 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:15.735175 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:15.735127 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:44:15.738172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:15.738148 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:44:22.271103 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:22.271066 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:32.271289 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:32.271249 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:36.095570 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:44:36.095532 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:44:41.638815 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:41.638761 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:44:42.271021 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:42.270981 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:44.919314 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:44.919278 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:44:45.150593 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:45.150564 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:44:52.271087 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:52.271048 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:44:55.502675 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:55.502637 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:44:55.607930 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:44:55.607899 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:02.271842 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:02.271743 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:45:02.930678 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:02.930645 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:03.147987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:03.147945 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:06.096535 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:06.096501 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:45:08.080445 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:08.080414 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:22.990632 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:22.990597 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:23.116472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:23.116433 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:24.365467 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.365428 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:24.471670 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.471638 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:24.487182 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.487151 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll"] Oct 31 07:45:24.490707 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.490689 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.504961 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.504939 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"invalid-s3-secret-d46a81\"" Oct 31 07:45:24.505085 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.505058 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-secondary-d46a81-kube-rbac-proxy-sar-config\"" Oct 31 07:45:24.516614 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.516587 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"invalid-s3-sa-d46a81-dockercfg-x7ppm\"" Oct 31 07:45:24.516708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.516676 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-secondary-d46a81-predictor-serving-cert\"" Oct 31 07:45:24.517987 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.517969 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll"] Oct 31 07:45:24.557816 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.557758 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.557988 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.557828 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.557988 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.557911 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r775j\" (UniqueName: \"kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.557988 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.557976 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.558109 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.558060 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659165 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659202 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659219 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r775j\" (UniqueName: \"kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659245 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659295 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659707 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659686 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.659954 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.659935 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.660051 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.660034 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.661741 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.661722 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.682004 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.681982 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r775j\" (UniqueName: \"kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j\") pod \"isvc-secondary-d46a81-predictor-6949847f59-m6vll\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:24.807471 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:24.807439 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:25.114657 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:25.114614 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll"] Oct 31 07:45:25.117277 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:45:25.117240 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f152f3_4fde_45a9_97ae_928b4255a5e3.slice/crio-0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962 WatchSource:0}: Error finding container 0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962: Status 404 returned error can't find the container with id 0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962 Oct 31 07:45:25.675588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:25.675545 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" event={"ID":"84f152f3-4fde-45a9-97ae-928b4255a5e3","Type":"ContainerStarted","Data":"2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39"} Oct 31 07:45:25.675998 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:25.675600 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" event={"ID":"84f152f3-4fde-45a9-97ae-928b4255a5e3","Type":"ContainerStarted","Data":"0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962"} Oct 31 07:45:28.694210 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:28.694178 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/0.log" Oct 31 07:45:28.694585 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:28.694217 4870 generic.go:356] "Generic (PLEG): container finished" podID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerID="2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39" exitCode=1 Oct 31 07:45:28.694585 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:28.694308 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" event={"ID":"84f152f3-4fde-45a9-97ae-928b4255a5e3","Type":"ContainerDied","Data":"2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39"} Oct 31 07:45:29.699556 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:29.699530 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/0.log" Oct 31 07:45:29.699953 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:29.699642 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" event={"ID":"84f152f3-4fde-45a9-97ae-928b4255a5e3","Type":"ContainerStarted","Data":"3c14741b113a1ac6e63407ef0569bb4f69dfaed8a081b91c56a0f65811019ded"} Oct 31 07:45:32.713245 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.713215 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/1.log" Oct 31 07:45:32.713698 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.713652 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/0.log" Oct 31 07:45:32.713769 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.713708 4870 generic.go:356] "Generic (PLEG): container finished" podID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerID="3c14741b113a1ac6e63407ef0569bb4f69dfaed8a081b91c56a0f65811019ded" exitCode=1 Oct 31 07:45:32.713834 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.713774 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" event={"ID":"84f152f3-4fde-45a9-97ae-928b4255a5e3","Type":"ContainerDied","Data":"3c14741b113a1ac6e63407ef0569bb4f69dfaed8a081b91c56a0f65811019ded"} Oct 31 07:45:32.713896 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.713861 4870 scope.go:122] "RemoveContainer" containerID="2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39" Oct 31 07:45:32.714216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.714198 4870 scope.go:122] "RemoveContainer" containerID="2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39" Oct 31 07:45:32.724295 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:32.724273 4870 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-d46a81-predictor-6949847f59-m6vll_e2e-gw1-7281997251(84f152f3-4fde-45a9-97ae-928b4255a5e3)\"" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" Oct 31 07:45:32.724538 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:32.724513 4870 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d46a81-predictor-6949847f59-m6vll_e2e-gw1-7281997251_84f152f3-4fde-45a9-97ae-928b4255a5e3_0 in pod sandbox 0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962 from index: no such id: '2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39'" containerID="2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39" Oct 31 07:45:32.724592 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:32.724548 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-d46a81-predictor-6949847f59-m6vll_e2e-gw1-7281997251_84f152f3-4fde-45a9-97ae-928b4255a5e3_0 in pod sandbox 0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962 from index: no such id: '2380174c965e7c068ea8900186b33e46c44496074c7a0e123b2a6a3b6edcdf39'" Oct 31 07:45:33.720389 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:33.720357 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/1.log" Oct 31 07:45:36.096227 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:36.096192 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:45:40.519183 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.519148 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll"] Oct 31 07:45:40.564572 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.564542 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:40.602756 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.602723 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg"] Oct 31 07:45:40.603143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.603092 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" containerID="cri-o://85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1" gracePeriod=30 Oct 31 07:45:40.603426 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.603179 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kube-rbac-proxy" containerID="cri-o://3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443" gracePeriod=30 Oct 31 07:45:40.666346 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.666320 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:40.727992 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.727973 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/1.log" Oct 31 07:45:40.730065 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.730050 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:40.748986 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.748966 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-secondary-d46a81-predictor-6949847f59-m6vll_84f152f3-4fde-45a9-97ae-928b4255a5e3/storage-initializer/1.log" Oct 31 07:45:40.749076 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.749046 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" Oct 31 07:45:40.749076 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.749062 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll" event={"ID":"84f152f3-4fde-45a9-97ae-928b4255a5e3","Type":"ContainerDied","Data":"0c9108a6d2e97ff183bb823f86fa58931bee4a3df7155dfc1122f11d74fe0962"} Oct 31 07:45:40.749188 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.749121 4870 scope.go:122] "RemoveContainer" containerID="3c14741b113a1ac6e63407ef0569bb4f69dfaed8a081b91c56a0f65811019ded" Oct 31 07:45:40.751221 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.751197 4870 generic.go:356] "Generic (PLEG): container finished" podID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerID="3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443" exitCode=0 Oct 31 07:45:40.751318 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.751272 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerDied","Data":"3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443"} Oct 31 07:45:40.791713 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.791691 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location\") pod \"84f152f3-4fde-45a9-97ae-928b4255a5e3\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " Oct 31 07:45:40.791846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.791726 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config\") pod \"84f152f3-4fde-45a9-97ae-928b4255a5e3\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " Oct 31 07:45:40.791846 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.791829 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls\") pod \"84f152f3-4fde-45a9-97ae-928b4255a5e3\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " Oct 31 07:45:40.791975 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.791955 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j\" (UniqueName: \"kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j\") pod \"84f152f3-4fde-45a9-97ae-928b4255a5e3\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " Oct 31 07:45:40.792066 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.791963 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location" pod "84f152f3-4fde-45a9-97ae-928b4255a5e3" (UID: "84f152f3-4fde-45a9-97ae-928b4255a5e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:45:40.792108 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.792056 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert\") pod \"84f152f3-4fde-45a9-97ae-928b4255a5e3\" (UID: \"84f152f3-4fde-45a9-97ae-928b4255a5e3\") " Oct 31 07:45:40.792166 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.792148 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config" pod "84f152f3-4fde-45a9-97ae-928b4255a5e3" (UID: "84f152f3-4fde-45a9-97ae-928b4255a5e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:45:40.792337 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.792313 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert" pod "84f152f3-4fde-45a9-97ae-928b4255a5e3" (UID: "84f152f3-4fde-45a9-97ae-928b4255a5e3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:45:40.792466 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.792416 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/84f152f3-4fde-45a9-97ae-928b4255a5e3-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:40.792583 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.792562 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:40.793862 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.793841 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j" pod "84f152f3-4fde-45a9-97ae-928b4255a5e3" (UID: "84f152f3-4fde-45a9-97ae-928b4255a5e3"). InnerVolumeSpecName "kube-api-access-r775j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:45:40.793956 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.793942 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls" pod "84f152f3-4fde-45a9-97ae-928b4255a5e3" (UID: "84f152f3-4fde-45a9-97ae-928b4255a5e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:45:40.893514 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.893482 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/84f152f3-4fde-45a9-97ae-928b4255a5e3-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:40.893514 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.893509 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r775j\" (UniqueName: \"kubernetes.io/projected/84f152f3-4fde-45a9-97ae-928b4255a5e3-kube-api-access-r775j\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:40.893514 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:40.893519 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/84f152f3-4fde-45a9-97ae-928b4255a5e3-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:41.120238 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:41.120210 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll"] Oct 31 07:45:41.123983 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:41.123958 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-secondary-d46a81-predictor-6949847f59-m6vll"] Oct 31 07:45:41.661772 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:41.661733 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" path="/var/lib/kubelet/pods/84f152f3-4fde-45a9-97ae-928b4255a5e3/volumes" Oct 31 07:45:42.265551 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:42.265504 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.34:8643/healthz\": dial tcp 10.132.0.34:8643: connect: connection refused" Oct 31 07:45:42.270678 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:42.270644 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused" Oct 31 07:45:45.241667 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.241644 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:45:45.330264 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330182 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert\") pod \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " Oct 31 07:45:45.330264 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330256 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config\") pod \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " Oct 31 07:45:45.330485 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330289 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location\") pod \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " Oct 31 07:45:45.330485 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330320 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls\") pod \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " Oct 31 07:45:45.330485 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330436 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn\" (UniqueName: \"kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn\") pod \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\" (UID: \"8ed22fe2-e63a-4de8-a2c7-79c5cd366638\") " Oct 31 07:45:45.330628 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330581 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert" pod "8ed22fe2-e63a-4de8-a2c7-79c5cd366638" (UID: "8ed22fe2-e63a-4de8-a2c7-79c5cd366638"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:45:45.330628 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330593 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location" pod "8ed22fe2-e63a-4de8-a2c7-79c5cd366638" (UID: "8ed22fe2-e63a-4de8-a2c7-79c5cd366638"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:45:45.330706 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330632 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config" pod "8ed22fe2-e63a-4de8-a2c7-79c5cd366638" (UID: "8ed22fe2-e63a-4de8-a2c7-79c5cd366638"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:45:45.330770 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330745 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:45.330770 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330758 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:45.330896 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.330769 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:45.332387 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.332363 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls" pod "8ed22fe2-e63a-4de8-a2c7-79c5cd366638" (UID: "8ed22fe2-e63a-4de8-a2c7-79c5cd366638"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:45:45.332501 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.332429 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn" pod "8ed22fe2-e63a-4de8-a2c7-79c5cd366638" (UID: "8ed22fe2-e63a-4de8-a2c7-79c5cd366638"). InnerVolumeSpecName "kube-api-access-lczpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:45:45.431396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.431362 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:45.431396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.431388 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczpn\" (UniqueName: \"kubernetes.io/projected/8ed22fe2-e63a-4de8-a2c7-79c5cd366638-kube-api-access-lczpn\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:45.773935 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.773901 4870 generic.go:356] "Generic (PLEG): container finished" podID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerID="85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1" exitCode=0 Oct 31 07:45:45.774128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.773960 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" Oct 31 07:45:45.774128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.773996 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerDied","Data":"85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1"} Oct 31 07:45:45.774128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.774069 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg" event={"ID":"8ed22fe2-e63a-4de8-a2c7-79c5cd366638","Type":"ContainerDied","Data":"09eef8ffafa164df75ee5191347eed154ae2728d668551dc182414c7f41fec96"} Oct 31 07:45:45.774128 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.774104 4870 scope.go:122] "RemoveContainer" containerID="3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443" Oct 31 07:45:45.784013 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.783995 4870 scope.go:122] "RemoveContainer" containerID="85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1" Oct 31 07:45:45.791597 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.791581 4870 scope.go:122] "RemoveContainer" containerID="36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13" Oct 31 07:45:45.798674 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.798655 4870 scope.go:122] "RemoveContainer" containerID="3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443" Oct 31 07:45:45.798951 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:45.798933 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443\": container with ID starting with 3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443 not found: ID does not exist" containerID="3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443" Oct 31 07:45:45.799005 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.798959 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443"} err="failed to get container status \"3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443\": rpc error: code = NotFound desc = could not find container \"3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443\": container with ID starting with 3b58fabf238178a8f8c5d53a6b03eeb968dfa588a78fccdb7efdf61d2cd2d443 not found: ID does not exist" Oct 31 07:45:45.799005 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.798977 4870 scope.go:122] "RemoveContainer" containerID="85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1" Oct 31 07:45:45.799229 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:45.799211 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1\": container with ID starting with 85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1 not found: ID does not exist" containerID="85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1" Oct 31 07:45:45.799271 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.799238 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1"} err="failed to get container status \"85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1\": rpc error: code = NotFound desc = could not find container \"85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1\": container with ID starting with 85b6160fd86ce9b3c4d0ba9019d7972ce52fe9859c6fec46a8910345dfb0c2f1 not found: ID does not exist" Oct 31 07:45:45.799271 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.799254 4870 scope.go:122] "RemoveContainer" containerID="36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13" Oct 31 07:45:45.799515 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:45.799495 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13\": container with ID starting with 36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13 not found: ID does not exist" containerID="36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13" Oct 31 07:45:45.799554 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.799520 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13"} err="failed to get container status \"36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13\": rpc error: code = NotFound desc = could not find container \"36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13\": container with ID starting with 36022f4b622716fdba6a9d59a53ccdeb9127df596e17568938a47a38601e9a13 not found: ID does not exist" Oct 31 07:45:45.824079 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.824050 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg"] Oct 31 07:45:45.827661 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:45.827640 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-primary-d46a81-predictor-7c5bfc86d4-6frqg"] Oct 31 07:45:47.660928 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:47.660897 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" path="/var/lib/kubelet/pods/8ed22fe2-e63a-4de8-a2c7-79c5cd366638/volumes" Oct 31 07:45:48.286162 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.286127 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:48.390243 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.390212 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:48.406262 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406236 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q"] Oct 31 07:45:48.406613 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406595 4870 cpu_manager.go:414] "containerMap: removing container" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" Oct 31 07:45:48.406693 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406615 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" Oct 31 07:45:48.406693 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406631 4870 cpu_manager.go:414] "containerMap: removing container" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerName="storage-initializer" Oct 31 07:45:48.406693 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406639 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerName="storage-initializer" Oct 31 07:45:48.406693 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406686 4870 cpu_manager.go:414] "containerMap: removing container" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerName="storage-initializer" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406695 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerName="storage-initializer" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406709 4870 cpu_manager.go:414] "containerMap: removing container" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="storage-initializer" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406717 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="storage-initializer" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406732 4870 cpu_manager.go:414] "containerMap: removing container" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kube-rbac-proxy" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406739 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kube-rbac-proxy" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406839 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerName="storage-initializer" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406854 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kube-rbac-proxy" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406866 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ed22fe2-e63a-4de8-a2c7-79c5cd366638" containerName="kserve-container" Oct 31 07:45:48.406923 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.406875 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f152f3-4fde-45a9-97ae-928b4255a5e3" containerName="storage-initializer" Oct 31 07:45:48.411920 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.411900 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.426516 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.426497 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:45:48.438234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.438205 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:45:48.438353 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.438273 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"fail-s3-secret-3ace72\"" Oct 31 07:45:48.438353 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.438282 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-init-fail-3ace72-kube-rbac-proxy-sar-config\"" Oct 31 07:45:48.438353 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.438321 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"fail-s3-sa-3ace72-dockercfg-zktfr\"" Oct 31 07:45:48.438487 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.438374 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-init-fail-3ace72-predictor-serving-cert\"" Oct 31 07:45:48.443852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.438857 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:45:48.443852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.443718 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q"] Oct 31 07:45:48.556411 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.556375 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.556577 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.556432 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.556577 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.556535 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.556577 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.556569 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.556679 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.556598 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hsjq\" (UniqueName: \"kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.657610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.657576 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.657610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.657611 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.657890 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.657632 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hsjq\" (UniqueName: \"kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.657890 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.657755 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.657890 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.657833 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.658053 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.658037 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.658390 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.658361 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.658491 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.658401 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.660130 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.660111 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.680608 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.680582 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hsjq\" (UniqueName: \"kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq\") pod \"isvc-init-fail-3ace72-predictor-85864fb64c-l954q\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:48.728617 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:48.728585 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:49.034588 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:49.034562 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q"] Oct 31 07:45:49.036953 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:45:49.036923 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244124f3_2287_438a_b9d2_3c43572dc513.slice/crio-b7c21e3fbaa0dea325f9ef8200f8856bd93795473fffc803b4f64e1b9b1bc75d WatchSource:0}: Error finding container b7c21e3fbaa0dea325f9ef8200f8856bd93795473fffc803b4f64e1b9b1bc75d: Status 404 returned error can't find the container with id b7c21e3fbaa0dea325f9ef8200f8856bd93795473fffc803b4f64e1b9b1bc75d Oct 31 07:45:49.793186 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:49.793153 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" event={"ID":"244124f3-2287-438a-b9d2-3c43572dc513","Type":"ContainerStarted","Data":"f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0"} Oct 31 07:45:49.793551 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:49.793208 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" event={"ID":"244124f3-2287-438a-b9d2-3c43572dc513","Type":"ContainerStarted","Data":"b7c21e3fbaa0dea325f9ef8200f8856bd93795473fffc803b4f64e1b9b1bc75d"} Oct 31 07:45:51.803071 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:51.803044 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-init-fail-3ace72-predictor-85864fb64c-l954q_244124f3-2287-438a-b9d2-3c43572dc513/storage-initializer/0.log" Oct 31 07:45:51.803439 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:51.803081 4870 generic.go:356] "Generic (PLEG): container finished" podID="244124f3-2287-438a-b9d2-3c43572dc513" containerID="f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0" exitCode=1 Oct 31 07:45:51.803439 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:51.803125 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" event={"ID":"244124f3-2287-438a-b9d2-3c43572dc513","Type":"ContainerDied","Data":"f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0"} Oct 31 07:45:52.809353 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:52.809324 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-init-fail-3ace72-predictor-85864fb64c-l954q_244124f3-2287-438a-b9d2-3c43572dc513/storage-initializer/0.log" Oct 31 07:45:52.809730 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:52.809376 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" event={"ID":"244124f3-2287-438a-b9d2-3c43572dc513","Type":"ContainerStarted","Data":"27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035"} Oct 31 07:45:53.370063 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:53.370021 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q"] Oct 31 07:45:53.410434 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:53.410402 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:53.814097 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:53.814052 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" containerID="cri-o://27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035" gracePeriod=30 Oct 31 07:45:56.458313 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.458291 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-init-fail-3ace72-predictor-85864fb64c-l954q_244124f3-2287-438a-b9d2-3c43572dc513/storage-initializer/1.log" Oct 31 07:45:56.458723 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.458702 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-init-fail-3ace72-predictor-85864fb64c-l954q_244124f3-2287-438a-b9d2-3c43572dc513/storage-initializer/0.log" Oct 31 07:45:56.461141 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.461122 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:56.623845 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.623810 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls\" (UniqueName: \"kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls\") pod \"244124f3-2287-438a-b9d2-3c43572dc513\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " Oct 31 07:45:56.624015 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.623859 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location\") pod \"244124f3-2287-438a-b9d2-3c43572dc513\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " Oct 31 07:45:56.624015 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.623920 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert\") pod \"244124f3-2287-438a-b9d2-3c43572dc513\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " Oct 31 07:45:56.624015 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.623943 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq\" (UniqueName: \"kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq\") pod \"244124f3-2287-438a-b9d2-3c43572dc513\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " Oct 31 07:45:56.624015 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.623983 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config\") pod \"244124f3-2287-438a-b9d2-3c43572dc513\" (UID: \"244124f3-2287-438a-b9d2-3c43572dc513\") " Oct 31 07:45:56.624216 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.624190 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location" pod "244124f3-2287-438a-b9d2-3c43572dc513" (UID: "244124f3-2287-438a-b9d2-3c43572dc513"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:45:56.624343 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.624319 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert" pod "244124f3-2287-438a-b9d2-3c43572dc513" (UID: "244124f3-2287-438a-b9d2-3c43572dc513"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:45:56.624406 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.624384 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config" pod "244124f3-2287-438a-b9d2-3c43572dc513" (UID: "244124f3-2287-438a-b9d2-3c43572dc513"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:45:56.626049 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.626030 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls" pod "244124f3-2287-438a-b9d2-3c43572dc513" (UID: "244124f3-2287-438a-b9d2-3c43572dc513"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:45:56.626113 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.626031 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq" pod "244124f3-2287-438a-b9d2-3c43572dc513" (UID: "244124f3-2287-438a-b9d2-3c43572dc513"). InnerVolumeSpecName "kube-api-access-2hsjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:45:56.724753 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.724718 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:56.724753 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.724747 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/244124f3-2287-438a-b9d2-3c43572dc513-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:56.724753 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.724756 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/244124f3-2287-438a-b9d2-3c43572dc513-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:56.724992 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.724766 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/244124f3-2287-438a-b9d2-3c43572dc513-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:56.724992 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.724775 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hsjq\" (UniqueName: \"kubernetes.io/projected/244124f3-2287-438a-b9d2-3c43572dc513-kube-api-access-2hsjq\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:45:56.827839 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.827808 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-init-fail-3ace72-predictor-85864fb64c-l954q_244124f3-2287-438a-b9d2-3c43572dc513/storage-initializer/1.log" Oct 31 07:45:56.828240 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.828224 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw1-7281997251_isvc-init-fail-3ace72-predictor-85864fb64c-l954q_244124f3-2287-438a-b9d2-3c43572dc513/storage-initializer/0.log" Oct 31 07:45:56.828295 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.828260 4870 generic.go:356] "Generic (PLEG): container finished" podID="244124f3-2287-438a-b9d2-3c43572dc513" containerID="27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035" exitCode=1 Oct 31 07:45:56.828341 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.828326 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" Oct 31 07:45:56.828384 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.828346 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" event={"ID":"244124f3-2287-438a-b9d2-3c43572dc513","Type":"ContainerDied","Data":"27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035"} Oct 31 07:45:56.828427 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.828405 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q" event={"ID":"244124f3-2287-438a-b9d2-3c43572dc513","Type":"ContainerDied","Data":"b7c21e3fbaa0dea325f9ef8200f8856bd93795473fffc803b4f64e1b9b1bc75d"} Oct 31 07:45:56.828466 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.828432 4870 scope.go:122] "RemoveContainer" containerID="27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035" Oct 31 07:45:56.837419 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.837404 4870 scope.go:122] "RemoveContainer" containerID="f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0" Oct 31 07:45:56.846973 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.846955 4870 scope.go:122] "RemoveContainer" containerID="27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035" Oct 31 07:45:56.847366 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:56.847348 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035\": container with ID starting with 27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035 not found: ID does not exist" containerID="27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035" Oct 31 07:45:56.847420 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.847374 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035"} err="failed to get container status \"27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035\": rpc error: code = NotFound desc = could not find container \"27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035\": container with ID starting with 27a1ec426fb6d463deb2cb2c860f1be78b9cf10af0df190161b23ef95769e035 not found: ID does not exist" Oct 31 07:45:56.847420 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.847390 4870 scope.go:122] "RemoveContainer" containerID="f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0" Oct 31 07:45:56.847649 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:45:56.847633 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0\": container with ID starting with f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0 not found: ID does not exist" containerID="f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0" Oct 31 07:45:56.847704 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.847652 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0"} err="failed to get container status \"f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0\": rpc error: code = NotFound desc = could not find container \"f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0\": container with ID starting with f395add08f2ea570592cf785efc428ba02f261029ad6f88b9d4b415f0bc719a0 not found: ID does not exist" Oct 31 07:45:56.907716 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.907638 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q"] Oct 31 07:45:56.911329 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:56.911308 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-init-fail-3ace72-predictor-85864fb64c-l954q"] Oct 31 07:45:57.661013 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:57.660979 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244124f3-2287-438a-b9d2-3c43572dc513" path="/var/lib/kubelet/pods/244124f3-2287-438a-b9d2-3c43572dc513/volumes" Oct 31 07:45:59.040517 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.040485 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:59.164683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.164652 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf"] Oct 31 07:45:59.165099 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.165076 4870 cpu_manager.go:414] "containerMap: removing container" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" Oct 31 07:45:59.165161 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.165101 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" Oct 31 07:45:59.165161 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.165116 4870 cpu_manager.go:414] "containerMap: removing container" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" Oct 31 07:45:59.165161 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.165122 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" Oct 31 07:45:59.165260 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.165202 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" Oct 31 07:45:59.165260 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.165216 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="244124f3-2287-438a-b9d2-3c43572dc513" containerName="storage-initializer" Oct 31 07:45:59.169921 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.169900 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.184627 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.184605 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:45:59.196281 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.196257 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:45:59.196281 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.196268 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:45:59.196281 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.196279 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"raw-sklearn-011e1-predictor-serving-cert\"" Oct 31 07:45:59.196459 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.196300 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"raw-sklearn-011e1-kube-rbac-proxy-sar-config\"" Oct 31 07:45:59.196518 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.196487 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:45:59.196562 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.196517 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:45:59.197786 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.197769 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf"] Oct 31 07:45:59.200091 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.200072 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:45:59.344997 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.344916 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.344997 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.344976 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.345199 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.345077 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.345199 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.345129 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.345310 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.345236 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgw2g\" (UniqueName: \"kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.445670 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.445639 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgw2g\" (UniqueName: \"kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.445670 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.445670 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.445952 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.445700 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.445952 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.445740 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.445952 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.445781 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.446154 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.446131 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.446452 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.446435 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.446487 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.446451 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.448246 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.448230 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.468087 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.468063 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgw2g\" (UniqueName: \"kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g\") pod \"raw-sklearn-011e1-predictor-655fdccbdf-lmpnf\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.487669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.487645 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:45:59.793271 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.793112 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf"] Oct 31 07:45:59.796210 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:45:59.796184 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb2a5fa_00f8_48d4_be36_73bee94e453f.slice/crio-0f1173170ffaa2ef00c86206748e3035d366f4ea95d6d38f8ab9996cf4d81a1c WatchSource:0}: Error finding container 0f1173170ffaa2ef00c86206748e3035d366f4ea95d6d38f8ab9996cf4d81a1c: Status 404 returned error can't find the container with id 0f1173170ffaa2ef00c86206748e3035d366f4ea95d6d38f8ab9996cf4d81a1c Oct 31 07:45:59.843636 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:45:59.843604 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerStarted","Data":"0f1173170ffaa2ef00c86206748e3035d366f4ea95d6d38f8ab9996cf4d81a1c"} Oct 31 07:46:00.849186 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:00.849150 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerStarted","Data":"533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197"} Oct 31 07:46:03.863478 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:03.863446 4870 generic.go:356] "Generic (PLEG): container finished" podID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerID="533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197" exitCode=0 Oct 31 07:46:03.863873 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:03.863518 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerDied","Data":"533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197"} Oct 31 07:46:04.870179 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:04.870144 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerStarted","Data":"54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f"} Oct 31 07:46:04.870564 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:04.870206 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerStarted","Data":"00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf"} Oct 31 07:46:04.870564 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:04.870380 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:46:04.926019 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:04.925970 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podStartSLOduration=2.249249263 podStartE2EDuration="5.92595671s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.676707447s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:45:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:46:04.925060274 +0000 UTC m=+1009.397454523" watchObservedRunningTime="2026-10-31 07:46:04.92595671 +0000 UTC m=+1009.398350958" Oct 31 07:46:05.874731 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:05.874698 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:46:05.876161 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:05.876127 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:06.095349 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:46:06.095316 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:46:06.878435 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:06.878394 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:08.198492 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:08.198451 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:46:11.883602 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:11.883574 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:46:11.884168 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:11.884139 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:21.884223 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:21.884173 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:31.884561 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:31.884473 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:36.095828 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:46:36.095778 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:46:41.884144 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:41.884101 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:51.884814 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:51.884763 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:46:59.215350 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:59.215314 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:46:59.325041 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:46:59.325004 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:01.884514 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:01.884474 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:47:06.095258 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:47:06.095221 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:47:08.314100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:08.314045 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:11.884234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:11.884195 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:47:21.884955 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:21.884918 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:47:29.855905 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:29.855871 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf"] Oct 31 07:47:29.856407 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:29.856168 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" containerID="cri-o://00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf" gracePeriod=30 Oct 31 07:47:29.856407 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:29.856194 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kube-rbac-proxy" containerID="cri-o://54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f" gracePeriod=30 Oct 31 07:47:29.889690 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:29.889667 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:30.227268 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:30.227184 4870 generic.go:356] "Generic (PLEG): container finished" podID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerID="54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f" exitCode=0 Oct 31 07:47:30.227404 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:30.227256 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerDied","Data":"54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f"} Oct 31 07:47:31.006274 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:31.006239 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:31.197569 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:31.197503 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:31.878876 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:31.878832 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused" Oct 31 07:47:31.884207 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:31.884179 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused" Oct 31 07:47:34.703122 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.703093 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:47:34.789086 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789054 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls\") pod \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " Oct 31 07:47:34.789244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789099 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert\") pod \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " Oct 31 07:47:34.789244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789132 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config\") pod \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " Oct 31 07:47:34.789244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789186 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g\" (UniqueName: \"kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g\") pod \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " Oct 31 07:47:34.789244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789228 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location\") pod \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\" (UID: \"2cb2a5fa-00f8-48d4-be36-73bee94e453f\") " Oct 31 07:47:34.789597 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789571 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert" pod "2cb2a5fa-00f8-48d4-be36-73bee94e453f" (UID: "2cb2a5fa-00f8-48d4-be36-73bee94e453f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:47:34.789663 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789608 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location" pod "2cb2a5fa-00f8-48d4-be36-73bee94e453f" (UID: "2cb2a5fa-00f8-48d4-be36-73bee94e453f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:47:34.789663 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.789602 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config" pod "2cb2a5fa-00f8-48d4-be36-73bee94e453f" (UID: "2cb2a5fa-00f8-48d4-be36-73bee94e453f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:47:34.791221 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.791197 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g" pod "2cb2a5fa-00f8-48d4-be36-73bee94e453f" (UID: "2cb2a5fa-00f8-48d4-be36-73bee94e453f"). InnerVolumeSpecName "kube-api-access-qgw2g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:47:34.791275 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.791218 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls" pod "2cb2a5fa-00f8-48d4-be36-73bee94e453f" (UID: "2cb2a5fa-00f8-48d4-be36-73bee94e453f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:47:34.889736 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.889700 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2cb2a5fa-00f8-48d4-be36-73bee94e453f-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:47:34.889736 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.889730 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:47:34.889951 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.889748 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:47:34.889951 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.889761 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgw2g\" (UniqueName: \"kubernetes.io/projected/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kube-api-access-qgw2g\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:47:34.889951 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:34.889773 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2cb2a5fa-00f8-48d4-be36-73bee94e453f-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:47:35.249943 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.249858 4870 generic.go:356] "Generic (PLEG): container finished" podID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerID="00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf" exitCode=0 Oct 31 07:47:35.249943 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.249915 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" Oct 31 07:47:35.250121 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.249942 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerDied","Data":"00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf"} Oct 31 07:47:35.250121 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.250032 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf" event={"ID":"2cb2a5fa-00f8-48d4-be36-73bee94e453f","Type":"ContainerDied","Data":"0f1173170ffaa2ef00c86206748e3035d366f4ea95d6d38f8ab9996cf4d81a1c"} Oct 31 07:47:35.250121 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.250060 4870 scope.go:122] "RemoveContainer" containerID="54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f" Oct 31 07:47:35.258992 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.258976 4870 scope.go:122] "RemoveContainer" containerID="00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf" Oct 31 07:47:35.266109 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.266092 4870 scope.go:122] "RemoveContainer" containerID="533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197" Oct 31 07:47:35.273137 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.273114 4870 scope.go:122] "RemoveContainer" containerID="54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f" Oct 31 07:47:35.273372 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:47:35.273354 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f\": container with ID starting with 54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f not found: ID does not exist" containerID="54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f" Oct 31 07:47:35.273417 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.273381 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f"} err="failed to get container status \"54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f\": rpc error: code = NotFound desc = could not find container \"54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f\": container with ID starting with 54cba7f5b99543537b7365e856505e47051e071e5d7e4b108a335790aa440b2f not found: ID does not exist" Oct 31 07:47:35.273417 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.273399 4870 scope.go:122] "RemoveContainer" containerID="00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf" Oct 31 07:47:35.273605 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:47:35.273590 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf\": container with ID starting with 00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf not found: ID does not exist" containerID="00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf" Oct 31 07:47:35.273642 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.273610 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf"} err="failed to get container status \"00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf\": rpc error: code = NotFound desc = could not find container \"00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf\": container with ID starting with 00928a2890e01074d06785c3d890282038188c4c00add35fc5f45f4925f479cf not found: ID does not exist" Oct 31 07:47:35.273642 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.273621 4870 scope.go:122] "RemoveContainer" containerID="533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197" Oct 31 07:47:35.273827 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:47:35.273810 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197\": container with ID starting with 533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197 not found: ID does not exist" containerID="533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197" Oct 31 07:47:35.273873 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.273833 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197"} err="failed to get container status \"533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197\": rpc error: code = NotFound desc = could not find container \"533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197\": container with ID starting with 533d5e983e3045375c50e1e538c457528236117f466d4d4a4af0b76efbc62197 not found: ID does not exist" Oct 31 07:47:35.311447 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.311423 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf"] Oct 31 07:47:35.323217 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.323195 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-011e1-predictor-655fdccbdf-lmpnf"] Oct 31 07:47:35.661374 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:35.661344 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" path="/var/lib/kubelet/pods/2cb2a5fa-00f8-48d4-be36-73bee94e453f/volumes" Oct 31 07:47:36.095598 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:47:36.095565 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:47:37.309230 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.309187 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:37.425582 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425550 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w"] Oct 31 07:47:37.425949 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425931 4870 cpu_manager.go:414] "containerMap: removing container" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="storage-initializer" Oct 31 07:47:37.425949 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425949 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="storage-initializer" Oct 31 07:47:37.426111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425960 4870 cpu_manager.go:414] "containerMap: removing container" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" Oct 31 07:47:37.426111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425965 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" Oct 31 07:47:37.426111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425972 4870 cpu_manager.go:414] "containerMap: removing container" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kube-rbac-proxy" Oct 31 07:47:37.426111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.425976 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kube-rbac-proxy" Oct 31 07:47:37.426111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.426034 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kserve-container" Oct 31 07:47:37.426111 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.426043 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb2a5fa-00f8-48d4-be36-73bee94e453f" containerName="kube-rbac-proxy" Oct 31 07:47:37.430749 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.430730 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.444862 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.444834 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:47:37.456328 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.456304 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:47:37.456454 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.456340 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"raw-sklearn-runtime-2df8d-kube-rbac-proxy-sar-config\"" Oct 31 07:47:37.456454 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.456356 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"raw-sklearn-runtime-2df8d-predictor-serving-cert\"" Oct 31 07:47:37.456454 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.456377 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:47:37.456454 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.456379 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:47:37.456454 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.456397 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:47:37.457675 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.457659 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w"] Oct 31 07:47:37.509199 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.509165 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.509351 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.509230 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.509351 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.509299 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.509351 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.509343 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.509491 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.509366 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkfgf\" (UniqueName: \"kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.559142 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.559115 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:47:37.610565 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.610506 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.610565 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.610557 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.610719 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.610586 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.610719 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.610616 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.610719 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.610635 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkfgf\" (UniqueName: \"kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.611085 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.611061 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.611275 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.611260 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.611326 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.611308 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.613168 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.613149 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.633002 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.632982 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkfgf\" (UniqueName: \"kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf\") pod \"raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:37.747634 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:37.747599 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:38.055195 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:38.055170 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w"] Oct 31 07:47:38.057456 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:47:38.057422 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe32eac5_e10f_4f28_bc11_c99b7ec704bc.slice/crio-d8e83621f3451ea7190a8bc9eecb6e44261c72baaa7751977fed0f49f7e31a33 WatchSource:0}: Error finding container d8e83621f3451ea7190a8bc9eecb6e44261c72baaa7751977fed0f49f7e31a33: Status 404 returned error can't find the container with id d8e83621f3451ea7190a8bc9eecb6e44261c72baaa7751977fed0f49f7e31a33 Oct 31 07:47:38.264652 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:38.264615 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerStarted","Data":"4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a"} Oct 31 07:47:38.264864 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:38.264677 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerStarted","Data":"d8e83621f3451ea7190a8bc9eecb6e44261c72baaa7751977fed0f49f7e31a33"} Oct 31 07:47:42.282892 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:42.282856 4870 generic.go:356] "Generic (PLEG): container finished" podID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerID="4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a" exitCode=0 Oct 31 07:47:42.283309 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:42.282934 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerDied","Data":"4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a"} Oct 31 07:47:43.288694 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:43.288659 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerStarted","Data":"32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9"} Oct 31 07:47:43.289255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:43.288720 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerStarted","Data":"4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9"} Oct 31 07:47:43.289255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:43.289019 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:43.289255 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:43.289183 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:43.290602 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:43.290573 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:47:43.344610 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:43.344565 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podStartSLOduration=2.366791639 podStartE2EDuration="6.344552845s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977761206s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:47:43.343547587 +0000 UTC m=+1107.815941839" watchObservedRunningTime="2026-10-31 07:47:43.344552845 +0000 UTC m=+1107.816947091" Oct 31 07:47:44.293318 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:44.293278 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:47:49.297606 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:49.297577 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:47:49.298100 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:49.298071 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:47:59.298565 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:47:59.298478 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:48:06.095600 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:48:06.095558 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:48:08.468377 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:08.468345 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:48:09.298110 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:09.298071 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:48:19.298117 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:19.298073 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:48:29.298563 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:29.298523 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:48:36.096223 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:48:36.096182 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:48:39.298003 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:39.297957 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:48:49.299026 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:49.298979 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:48:59.299017 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:48:59.298979 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:49:06.096618 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:49:06.096577 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:49:08.269306 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.269266 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w"] Oct 31 07:49:08.269713 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.269596 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" containerID="cri-o://4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9" gracePeriod=30 Oct 31 07:49:08.269713 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.269638 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kube-rbac-proxy" containerID="cri-o://32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9" gracePeriod=30 Oct 31 07:49:08.316143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.316115 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:49:08.567090 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.567057 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:49:08.638430 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.638396 4870 generic.go:356] "Generic (PLEG): container finished" podID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerID="32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9" exitCode=0 Oct 31 07:49:08.638599 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:08.638449 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerDied","Data":"32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9"} Oct 31 07:49:09.294488 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:09.294450 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused" Oct 31 07:49:09.298819 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:09.298775 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused" Oct 31 07:49:13.210426 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.210403 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:49:13.261182 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261154 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls\") pod \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " Oct 31 07:49:13.261305 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261224 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config\") pod \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " Oct 31 07:49:13.261305 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261244 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf\" (UniqueName: \"kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf\") pod \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " Oct 31 07:49:13.261305 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261291 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location\") pod \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " Oct 31 07:49:13.261472 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261316 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert\") pod \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\" (UID: \"fe32eac5-e10f-4f28-bc11-c99b7ec704bc\") " Oct 31 07:49:13.261620 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261596 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location" pod "fe32eac5-e10f-4f28-bc11-c99b7ec704bc" (UID: "fe32eac5-e10f-4f28-bc11-c99b7ec704bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:49:13.261683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261630 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config" pod "fe32eac5-e10f-4f28-bc11-c99b7ec704bc" (UID: "fe32eac5-e10f-4f28-bc11-c99b7ec704bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:49:13.261721 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.261672 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert" pod "fe32eac5-e10f-4f28-bc11-c99b7ec704bc" (UID: "fe32eac5-e10f-4f28-bc11-c99b7ec704bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:49:13.263151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.263131 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls" pod "fe32eac5-e10f-4f28-bc11-c99b7ec704bc" (UID: "fe32eac5-e10f-4f28-bc11-c99b7ec704bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:49:13.263343 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.263321 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf" pod "fe32eac5-e10f-4f28-bc11-c99b7ec704bc" (UID: "fe32eac5-e10f-4f28-bc11-c99b7ec704bc"). InnerVolumeSpecName "kube-api-access-fkfgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:49:13.362129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.362050 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:49:13.362129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.362076 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:49:13.362129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.362087 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkfgf\" (UniqueName: \"kubernetes.io/projected/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kube-api-access-fkfgf\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:49:13.362129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.362096 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:49:13.362129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.362104 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fe32eac5-e10f-4f28-bc11-c99b7ec704bc-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:49:13.667235 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.667149 4870 generic.go:356] "Generic (PLEG): container finished" podID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerID="4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9" exitCode=0 Oct 31 07:49:13.667372 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.667231 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" Oct 31 07:49:13.667372 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.667231 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerDied","Data":"4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9"} Oct 31 07:49:13.667372 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.667360 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w" event={"ID":"fe32eac5-e10f-4f28-bc11-c99b7ec704bc","Type":"ContainerDied","Data":"d8e83621f3451ea7190a8bc9eecb6e44261c72baaa7751977fed0f49f7e31a33"} Oct 31 07:49:13.667473 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.667387 4870 scope.go:122] "RemoveContainer" containerID="32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9" Oct 31 07:49:13.675627 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.675607 4870 scope.go:122] "RemoveContainer" containerID="4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9" Oct 31 07:49:13.683275 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.683253 4870 scope.go:122] "RemoveContainer" containerID="4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a" Oct 31 07:49:13.689878 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.689862 4870 scope.go:122] "RemoveContainer" containerID="32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9" Oct 31 07:49:13.690126 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:49:13.690108 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9\": container with ID starting with 32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9 not found: ID does not exist" containerID="32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9" Oct 31 07:49:13.690166 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.690134 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9"} err="failed to get container status \"32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9\": rpc error: code = NotFound desc = could not find container \"32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9\": container with ID starting with 32ba0deba1ae09b2f88c268637f1c7ffcb67edb87b239f3e3f57f7eeae2203c9 not found: ID does not exist" Oct 31 07:49:13.690166 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.690151 4870 scope.go:122] "RemoveContainer" containerID="4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9" Oct 31 07:49:13.690375 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:49:13.690358 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9\": container with ID starting with 4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9 not found: ID does not exist" containerID="4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9" Oct 31 07:49:13.690410 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.690381 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9"} err="failed to get container status \"4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9\": rpc error: code = NotFound desc = could not find container \"4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9\": container with ID starting with 4674bee54b40c126f3a3235abe9a28b318f3cbcf054d3484c84584d0177488d9 not found: ID does not exist" Oct 31 07:49:13.690410 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.690395 4870 scope.go:122] "RemoveContainer" containerID="4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a" Oct 31 07:49:13.690613 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:49:13.690590 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a\": container with ID starting with 4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a not found: ID does not exist" containerID="4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a" Oct 31 07:49:13.690694 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.690613 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a"} err="failed to get container status \"4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a\": rpc error: code = NotFound desc = could not find container \"4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a\": container with ID starting with 4934a17af13cbbf815570a789fd1cdd5a1078fe2683a6869a913bb1b5e45e44a not found: ID does not exist" Oct 31 07:49:13.717110 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.717089 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w"] Oct 31 07:49:13.728719 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:13.728695 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/raw-sklearn-runtime-2df8d-predictor-5d687c9965-75x4w"] Oct 31 07:49:15.647778 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.647747 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:49:15.661071 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.661048 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" path="/var/lib/kubelet/pods/fe32eac5-e10f-4f28-bc11-c99b7ec704bc/volumes" Oct 31 07:49:15.760517 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.760487 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:49:15.765405 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.765381 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:49:15.803265 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.803241 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:49:15.897130 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897098 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57"] Oct 31 07:49:15.897458 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897443 4870 cpu_manager.go:414] "containerMap: removing container" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" Oct 31 07:49:15.897507 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897460 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" Oct 31 07:49:15.897507 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897486 4870 cpu_manager.go:414] "containerMap: removing container" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="storage-initializer" Oct 31 07:49:15.897507 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897491 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="storage-initializer" Oct 31 07:49:15.897507 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897504 4870 cpu_manager.go:414] "containerMap: removing container" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kube-rbac-proxy" Oct 31 07:49:15.897628 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897509 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kube-rbac-proxy" Oct 31 07:49:15.897628 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897569 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kube-rbac-proxy" Oct 31 07:49:15.897628 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.897582 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe32eac5-e10f-4f28-bc11-c99b7ec704bc" containerName="kserve-container" Oct 31 07:49:15.902521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.902462 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:15.917968 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.917943 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:49:15.918854 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.918828 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:49:15.930242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.930219 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Oct 31 07:49:15.930395 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.930257 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:49:15.930497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.930287 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:49:15.930591 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.930351 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:49:15.930683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.930389 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-create-v2-predictor-serving-cert\"" Oct 31 07:49:15.931081 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.931053 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:49:15.932145 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.932121 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57"] Oct 31 07:49:15.980133 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.980098 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:15.980133 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.980136 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:15.980312 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.980218 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsz2\" (UniqueName: \"kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:15.980312 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.980256 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:15.980312 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:15.980277 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.050080 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.050047 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh"] Oct 31 07:49:16.053888 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.053869 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.077927 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.077900 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh"] Oct 31 07:49:16.080863 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.080838 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.080968 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.080914 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.080968 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.080947 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.081079 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.081021 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsz2\" (UniqueName: \"kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.081079 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.081050 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.081385 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.081364 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.081631 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.081610 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.082077 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.082060 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.083537 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.083515 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.105164 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.105144 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsz2\" (UniqueName: \"kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2\") pod \"isvc-canary-create-v2-predictor-75ffb768cf-k8t57\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.119261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.119242 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-create-predictor-serving-cert\"" Oct 31 07:49:16.181913 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.181844 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.181913 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.181882 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.182101 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.181914 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.182101 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.181996 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.182101 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.182051 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddjv\" (UniqueName: \"kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.221453 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.221430 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:16.283538 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.283363 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.284011 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.283982 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.284143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.284054 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.284143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.284082 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.284143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.284108 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddjv\" (UniqueName: \"kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.284440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.284411 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.285032 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.284639 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.285138 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.285111 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.288524 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.287330 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.307009 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.306983 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddjv\" (UniqueName: \"kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv\") pod \"isvc-canary-create-predictor-66d784d8dd-j5lkh\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.371023 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.370970 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:16.525781 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.525755 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57"] Oct 31 07:49:16.528042 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:49:16.528011 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25dd474d_8099_4817_b277_d12d53fd0c35.slice/crio-697c52b95c5a210c5c38baa82d21818d002bef9b665567ae54a256a8f750e8de WatchSource:0}: Error finding container 697c52b95c5a210c5c38baa82d21818d002bef9b665567ae54a256a8f750e8de: Status 404 returned error can't find the container with id 697c52b95c5a210c5c38baa82d21818d002bef9b665567ae54a256a8f750e8de Oct 31 07:49:16.666324 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.666294 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh"] Oct 31 07:49:16.669373 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:49:16.669342 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19471ac5_8588_4a69_bcca_5e31e6cda521.slice/crio-8cb5f44cefcc3fdc3f641d07235080ad0806ade7600aac0fbcb46cc868e52455 WatchSource:0}: Error finding container 8cb5f44cefcc3fdc3f641d07235080ad0806ade7600aac0fbcb46cc868e52455: Status 404 returned error can't find the container with id 8cb5f44cefcc3fdc3f641d07235080ad0806ade7600aac0fbcb46cc868e52455 Oct 31 07:49:16.682444 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.682414 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerStarted","Data":"8cb5f44cefcc3fdc3f641d07235080ad0806ade7600aac0fbcb46cc868e52455"} Oct 31 07:49:16.684146 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.684118 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerStarted","Data":"78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495"} Oct 31 07:49:16.684235 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:16.684188 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerStarted","Data":"697c52b95c5a210c5c38baa82d21818d002bef9b665567ae54a256a8f750e8de"} Oct 31 07:49:17.688990 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:17.688948 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerStarted","Data":"93605fdadbd4ea20273fbbf655054801e9b6a2b62205b2b336cfba4e238cdc03"} Oct 31 07:49:20.708520 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:20.708485 4870 generic.go:356] "Generic (PLEG): container finished" podID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerID="93605fdadbd4ea20273fbbf655054801e9b6a2b62205b2b336cfba4e238cdc03" exitCode=0 Oct 31 07:49:20.709010 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:20.708521 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerDied","Data":"93605fdadbd4ea20273fbbf655054801e9b6a2b62205b2b336cfba4e238cdc03"} Oct 31 07:49:20.710287 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:20.710234 4870 generic.go:356] "Generic (PLEG): container finished" podID="25dd474d-8099-4817-b277-d12d53fd0c35" containerID="78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495" exitCode=0 Oct 31 07:49:20.710287 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:20.710276 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerDied","Data":"78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495"} Oct 31 07:49:21.716547 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.716511 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerStarted","Data":"549c0e26f180c8ad76807bdfa61b56b7da4acab183d867177c1440d634593957"} Oct 31 07:49:21.716983 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.716587 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerStarted","Data":"a5450bf289ca22c95068e10369893008424c513533b3b7f9b3db6974dd36525e"} Oct 31 07:49:21.716983 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.716833 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:21.718861 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.718829 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerStarted","Data":"d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386"} Oct 31 07:49:21.718980 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.718886 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerStarted","Data":"14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376"} Oct 31 07:49:21.719148 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.719130 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:21.719215 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.719171 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:21.720514 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.720490 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:49:21.760047 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.760004 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podStartSLOduration=2.002580546 podStartE2EDuration="5.759991858s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.757411312s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:49:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:49:21.759695835 +0000 UTC m=+1206.232090084" watchObservedRunningTime="2026-10-31 07:49:21.759991858 +0000 UTC m=+1206.232386106" Oct 31 07:49:21.802054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:21.802007 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podStartSLOduration=2.73208992 podStartE2EDuration="6.801995517s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.069905597s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:49:21.801382366 +0000 UTC m=+1206.273776623" watchObservedRunningTime="2026-10-31 07:49:21.801995517 +0000 UTC m=+1206.274389765" Oct 31 07:49:22.723711 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:22.723670 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:49:22.724143 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:22.723779 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:22.725039 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:22.725012 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:49:23.727671 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:23.727630 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:49:27.728186 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:27.728114 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:49:27.728555 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:27.728524 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:49:28.732916 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:28.732887 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:49:28.733362 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:28.733336 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:49:36.095945 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:49:36.095909 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:49:37.468499 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:37.468462 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:49:37.728697 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:37.728609 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:49:38.733479 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:38.733441 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:49:47.729525 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:47.729489 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:49:48.734274 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:48.734233 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:49:57.729560 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:57.729517 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:49:58.733969 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:49:58.733928 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:50:06.095733 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:50:06.095691 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:50:07.729473 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:07.729435 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:50:08.724503 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:08.724471 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:08.734089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:08.734060 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:50:17.728938 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:17.728894 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:50:18.734138 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:18.734093 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:50:22.640004 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:22.639967 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:27.728700 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:27.728663 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:50:28.733319 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:28.733280 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:50:35.440504 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:35.440473 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:36.095891 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:50:36.095856 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:50:37.729483 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:37.729446 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused" Oct 31 07:50:37.802231 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:37.802200 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:38.734365 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:38.734336 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:50:42.838512 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:42.838476 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:47.729412 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:47.729377 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:50:48.152219 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:48.152180 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh"] Oct 31 07:50:48.152446 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:48.152411 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" containerID="cri-o://a5450bf289ca22c95068e10369893008424c513533b3b7f9b3db6974dd36525e" gracePeriod=30 Oct 31 07:50:48.152536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:48.152455 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kube-rbac-proxy" containerID="cri-o://549c0e26f180c8ad76807bdfa61b56b7da4acab183d867177c1440d634593957" gracePeriod=30 Oct 31 07:50:48.728283 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:48.728239 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused" Oct 31 07:50:48.733592 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:48.733565 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused" Oct 31 07:50:49.084011 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:49.083978 4870 generic.go:356] "Generic (PLEG): container finished" podID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerID="549c0e26f180c8ad76807bdfa61b56b7da4acab183d867177c1440d634593957" exitCode=0 Oct 31 07:50:49.084200 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:49.084059 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerDied","Data":"549c0e26f180c8ad76807bdfa61b56b7da4acab183d867177c1440d634593957"} Oct 31 07:50:50.999214 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:50.999183 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57"] Oct 31 07:50:50.999666 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:50.999396 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" containerID="cri-o://14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376" gracePeriod=30 Oct 31 07:50:50.999666 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:50.999444 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kube-rbac-proxy" containerID="cri-o://d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386" gracePeriod=30 Oct 31 07:50:51.041494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:51.041465 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:52.098863 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:52.098829 4870 generic.go:356] "Generic (PLEG): container finished" podID="25dd474d-8099-4817-b277-d12d53fd0c35" containerID="d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386" exitCode=0 Oct 31 07:50:52.099234 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:52.098898 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerDied","Data":"d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386"} Oct 31 07:50:52.724467 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:52.724426 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": dial tcp 10.132.0.39:8643: connect: connection refused" Oct 31 07:50:53.105675 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.105640 4870 generic.go:356] "Generic (PLEG): container finished" podID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerID="a5450bf289ca22c95068e10369893008424c513533b3b7f9b3db6974dd36525e" exitCode=0 Oct 31 07:50:53.106081 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.105713 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerDied","Data":"a5450bf289ca22c95068e10369893008424c513533b3b7f9b3db6974dd36525e"} Oct 31 07:50:53.385650 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.385625 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:50:53.482812 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.482765 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config\") pod \"19471ac5-8588-4a69-bcca-5e31e6cda521\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " Oct 31 07:50:53.483001 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.482828 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls\" (UniqueName: \"kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls\") pod \"19471ac5-8588-4a69-bcca-5e31e6cda521\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " Oct 31 07:50:53.483001 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.482877 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location\") pod \"19471ac5-8588-4a69-bcca-5e31e6cda521\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " Oct 31 07:50:53.483001 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.482925 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv\" (UniqueName: \"kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv\") pod \"19471ac5-8588-4a69-bcca-5e31e6cda521\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " Oct 31 07:50:53.483001 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.482950 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert\") pod \"19471ac5-8588-4a69-bcca-5e31e6cda521\" (UID: \"19471ac5-8588-4a69-bcca-5e31e6cda521\") " Oct 31 07:50:53.483229 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.483187 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config" pod "19471ac5-8588-4a69-bcca-5e31e6cda521" (UID: "19471ac5-8588-4a69-bcca-5e31e6cda521"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:50:53.483321 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.483220 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location" pod "19471ac5-8588-4a69-bcca-5e31e6cda521" (UID: "19471ac5-8588-4a69-bcca-5e31e6cda521"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:50:53.483427 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.483370 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert" pod "19471ac5-8588-4a69-bcca-5e31e6cda521" (UID: "19471ac5-8588-4a69-bcca-5e31e6cda521"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:50:53.485091 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.485066 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv" pod "19471ac5-8588-4a69-bcca-5e31e6cda521" (UID: "19471ac5-8588-4a69-bcca-5e31e6cda521"). InnerVolumeSpecName "kube-api-access-mddjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:50:53.485159 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.485070 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls" pod "19471ac5-8588-4a69-bcca-5e31e6cda521" (UID: "19471ac5-8588-4a69-bcca-5e31e6cda521"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:50:53.583576 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.583542 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:53.583576 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.583573 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19471ac5-8588-4a69-bcca-5e31e6cda521-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:53.583576 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.583585 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19471ac5-8588-4a69-bcca-5e31e6cda521-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:53.583836 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.583594 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mddjv\" (UniqueName: \"kubernetes.io/projected/19471ac5-8588-4a69-bcca-5e31e6cda521-kube-api-access-mddjv\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:53.583836 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:53.583604 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19471ac5-8588-4a69-bcca-5e31e6cda521-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:54.111807 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.111755 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" event={"ID":"19471ac5-8588-4a69-bcca-5e31e6cda521","Type":"ContainerDied","Data":"8cb5f44cefcc3fdc3f641d07235080ad0806ade7600aac0fbcb46cc868e52455"} Oct 31 07:50:54.112224 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.111838 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh" Oct 31 07:50:54.112224 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.111842 4870 scope.go:122] "RemoveContainer" containerID="549c0e26f180c8ad76807bdfa61b56b7da4acab183d867177c1440d634593957" Oct 31 07:50:54.120278 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.120255 4870 scope.go:122] "RemoveContainer" containerID="a5450bf289ca22c95068e10369893008424c513533b3b7f9b3db6974dd36525e" Oct 31 07:50:54.128068 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.128050 4870 scope.go:122] "RemoveContainer" containerID="93605fdadbd4ea20273fbbf655054801e9b6a2b62205b2b336cfba4e238cdc03" Oct 31 07:50:54.173771 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.173744 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh"] Oct 31 07:50:54.179037 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:54.179017 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-predictor-66d784d8dd-j5lkh"] Oct 31 07:50:55.662545 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:55.662512 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" path="/var/lib/kubelet/pods/19471ac5-8588-4a69-bcca-5e31e6cda521/volumes" Oct 31 07:50:56.041676 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.041649 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:50:56.123177 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.123145 4870 generic.go:356] "Generic (PLEG): container finished" podID="25dd474d-8099-4817-b277-d12d53fd0c35" containerID="14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376" exitCode=0 Oct 31 07:50:56.123334 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.123198 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" Oct 31 07:50:56.123334 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.123203 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerDied","Data":"14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376"} Oct 31 07:50:56.123334 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.123254 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57" event={"ID":"25dd474d-8099-4817-b277-d12d53fd0c35","Type":"ContainerDied","Data":"697c52b95c5a210c5c38baa82d21818d002bef9b665567ae54a256a8f750e8de"} Oct 31 07:50:56.123334 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.123282 4870 scope.go:122] "RemoveContainer" containerID="d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386" Oct 31 07:50:56.131217 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.131196 4870 scope.go:122] "RemoveContainer" containerID="14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376" Oct 31 07:50:56.138504 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.138487 4870 scope.go:122] "RemoveContainer" containerID="78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495" Oct 31 07:50:56.145911 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.145890 4870 scope.go:122] "RemoveContainer" containerID="d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386" Oct 31 07:50:56.146158 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:50:56.146138 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386\": container with ID starting with d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386 not found: ID does not exist" containerID="d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386" Oct 31 07:50:56.146236 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.146170 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386"} err="failed to get container status \"d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386\": rpc error: code = NotFound desc = could not find container \"d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386\": container with ID starting with d8d5fdc8a30c76eb83e72cc6e88048ed91bc0dbfbb5af482bd35529c8d693386 not found: ID does not exist" Oct 31 07:50:56.146236 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.146198 4870 scope.go:122] "RemoveContainer" containerID="14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376" Oct 31 07:50:56.146428 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:50:56.146412 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376\": container with ID starting with 14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376 not found: ID does not exist" containerID="14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376" Oct 31 07:50:56.146468 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.146436 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376"} err="failed to get container status \"14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376\": rpc error: code = NotFound desc = could not find container \"14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376\": container with ID starting with 14707c5260ff650c37f0db30592d576fc7dbde53618565ed0d8454420a61f376 not found: ID does not exist" Oct 31 07:50:56.146468 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.146451 4870 scope.go:122] "RemoveContainer" containerID="78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495" Oct 31 07:50:56.146673 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:50:56.146655 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495\": container with ID starting with 78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495 not found: ID does not exist" containerID="78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495" Oct 31 07:50:56.146727 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.146677 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495"} err="failed to get container status \"78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495\": rpc error: code = NotFound desc = could not find container \"78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495\": container with ID starting with 78bae1e3661c0cc54091aeb21d091237f6442b04a5a9e4d533ceac528a77e495 not found: ID does not exist" Oct 31 07:50:56.205948 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.205906 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location\") pod \"25dd474d-8099-4817-b277-d12d53fd0c35\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " Oct 31 07:50:56.206129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.205980 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls\" (UniqueName: \"kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls\") pod \"25dd474d-8099-4817-b277-d12d53fd0c35\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " Oct 31 07:50:56.206129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.206008 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert\") pod \"25dd474d-8099-4817-b277-d12d53fd0c35\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " Oct 31 07:50:56.206129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.206044 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config\") pod \"25dd474d-8099-4817-b277-d12d53fd0c35\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " Oct 31 07:50:56.206129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.206101 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2\" (UniqueName: \"kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2\") pod \"25dd474d-8099-4817-b277-d12d53fd0c35\" (UID: \"25dd474d-8099-4817-b277-d12d53fd0c35\") " Oct 31 07:50:56.206348 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.206257 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location" pod "25dd474d-8099-4817-b277-d12d53fd0c35" (UID: "25dd474d-8099-4817-b277-d12d53fd0c35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:50:56.206389 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.206360 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert" pod "25dd474d-8099-4817-b277-d12d53fd0c35" (UID: "25dd474d-8099-4817-b277-d12d53fd0c35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:50:56.206424 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.206400 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config" pod "25dd474d-8099-4817-b277-d12d53fd0c35" (UID: "25dd474d-8099-4817-b277-d12d53fd0c35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:50:56.208054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.208030 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2" pod "25dd474d-8099-4817-b277-d12d53fd0c35" (UID: "25dd474d-8099-4817-b277-d12d53fd0c35"). InnerVolumeSpecName "kube-api-access-fqsz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:50:56.208146 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.208076 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls" pod "25dd474d-8099-4817-b277-d12d53fd0c35" (UID: "25dd474d-8099-4817-b277-d12d53fd0c35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:50:56.307475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.307443 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:56.307475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.307474 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqsz2\" (UniqueName: \"kubernetes.io/projected/25dd474d-8099-4817-b277-d12d53fd0c35-kube-api-access-fqsz2\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:56.307669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.307484 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25dd474d-8099-4817-b277-d12d53fd0c35-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:56.307669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.307494 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25dd474d-8099-4817-b277-d12d53fd0c35-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:56.307669 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.307504 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25dd474d-8099-4817-b277-d12d53fd0c35-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:50:56.467190 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.467102 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57"] Oct 31 07:50:56.478780 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:56.478749 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-create-v2-predictor-75ffb768cf-k8t57"] Oct 31 07:50:57.661784 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:57.661698 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" path="/var/lib/kubelet/pods/25dd474d-8099-4817-b277-d12d53fd0c35/volumes" Oct 31 07:50:58.640549 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.640503 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:58.798802 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.798760 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:58.892091 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892009 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2"] Oct 31 07:50:58.892326 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892313 4870 cpu_manager.go:414] "containerMap: removing container" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kube-rbac-proxy" Oct 31 07:50:58.892368 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892327 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kube-rbac-proxy" Oct 31 07:50:58.892368 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892348 4870 cpu_manager.go:414] "containerMap: removing container" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="storage-initializer" Oct 31 07:50:58.892368 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892353 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="storage-initializer" Oct 31 07:50:58.892368 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892359 4870 cpu_manager.go:414] "containerMap: removing container" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" Oct 31 07:50:58.892368 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892364 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892374 4870 cpu_manager.go:414] "containerMap: removing container" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892378 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892392 4870 cpu_manager.go:414] "containerMap: removing container" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kube-rbac-proxy" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892397 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kube-rbac-proxy" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892409 4870 cpu_manager.go:414] "containerMap: removing container" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="storage-initializer" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892414 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="storage-initializer" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892461 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kserve-container" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892468 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kserve-container" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892475 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="19471ac5-8588-4a69-bcca-5e31e6cda521" containerName="kube-rbac-proxy" Oct 31 07:50:58.892550 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.892482 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="25dd474d-8099-4817-b277-d12d53fd0c35" containerName="kube-rbac-proxy" Oct 31 07:50:58.897051 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.897031 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:58.913184 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.913147 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"kube-root-ca.crt\"" Oct 31 07:50:58.913184 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.913148 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"odh-kserve-custom-ca-bundle\"" Oct 31 07:50:58.913184 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.913197 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"seaweedfs-s3-creds\"" Oct 31 07:50:58.916375 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.916349 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:50:58.924479 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.924459 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Oct 31 07:50:58.924479 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.924462 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-7281997251\"/\"openshift-service-ca.crt\"" Oct 31 07:50:58.924653 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.924638 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"default-dockercfg-t4zp9\"" Oct 31 07:50:58.924717 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.924684 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Oct 31 07:50:58.926201 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:58.926181 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2"] Oct 31 07:50:59.030913 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.030880 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.031073 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.030927 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.031073 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.031036 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.031174 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.031070 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.031174 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.031090 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf7z8\" (UniqueName: \"kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.057804 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.057660 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5"] Oct 31 07:50:59.061326 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.061309 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.084192 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.084165 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5"] Oct 31 07:50:59.113126 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.113095 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-7281997251\"/\"isvc-canary-promote-predictor-serving-cert\"" Oct 31 07:50:59.132516 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.132487 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.132626 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.132525 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.132626 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.132569 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.132626 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.132590 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.132626 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.132606 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf7z8\" (UniqueName: \"kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.133070 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.133042 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.133215 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.133186 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.133323 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.133218 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.135366 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.135342 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.155217 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.155156 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf7z8\" (UniqueName: \"kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8\") pod \"isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.213702 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.213675 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:50:59.233087 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.233057 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.233180 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.233097 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcq6h\" (UniqueName: \"kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.233180 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.233129 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.233317 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.233297 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.233360 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.233335 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334011 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.333980 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334180 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334018 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334180 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334035 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334180 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334153 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcq6h\" (UniqueName: \"kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334350 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334192 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334561 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334530 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334684 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334630 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.334684 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.334669 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.336414 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.336396 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.356683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.356663 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcq6h\" (UniqueName: \"kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h\") pod \"isvc-canary-promote-predictor-574fc57cbd-r2ck5\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.378637 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.378614 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:50:59.519599 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.519574 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2"] Oct 31 07:50:59.521851 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:50:59.521821 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6958588f_c618_48c0_9315_954e82b8561c.slice/crio-ffc2b06eb014e44495c5feb77fc42cce2d79e8e7fa79e9c1a0210d182c5a1c6e WatchSource:0}: Error finding container ffc2b06eb014e44495c5feb77fc42cce2d79e8e7fa79e9c1a0210d182c5a1c6e: Status 404 returned error can't find the container with id ffc2b06eb014e44495c5feb77fc42cce2d79e8e7fa79e9c1a0210d182c5a1c6e Oct 31 07:50:59.674174 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:50:59.674136 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5"] Oct 31 07:50:59.677159 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:50:59.677081 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod554e5562_bed4_4c20_aff8_1d2e2ec57462.slice/crio-bea6f411cdb106c100fb305f4f1eef524268a8731e3a1a2b6d8c9fc73669a394 WatchSource:0}: Error finding container bea6f411cdb106c100fb305f4f1eef524268a8731e3a1a2b6d8c9fc73669a394: Status 404 returned error can't find the container with id bea6f411cdb106c100fb305f4f1eef524268a8731e3a1a2b6d8c9fc73669a394 Oct 31 07:51:00.143308 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:00.143270 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerStarted","Data":"918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332"} Oct 31 07:51:00.143820 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:00.143348 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerStarted","Data":"ffc2b06eb014e44495c5feb77fc42cce2d79e8e7fa79e9c1a0210d182c5a1c6e"} Oct 31 07:51:00.144912 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:00.144877 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerStarted","Data":"02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a"} Oct 31 07:51:00.145026 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:00.144939 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerStarted","Data":"bea6f411cdb106c100fb305f4f1eef524268a8731e3a1a2b6d8c9fc73669a394"} Oct 31 07:51:04.167892 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:04.167858 4870 generic.go:356] "Generic (PLEG): container finished" podID="6958588f-c618-48c0-9315-954e82b8561c" containerID="918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332" exitCode=0 Oct 31 07:51:04.168378 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:04.167935 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerDied","Data":"918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332"} Oct 31 07:51:04.169760 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:04.169729 4870 generic.go:356] "Generic (PLEG): container finished" podID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerID="02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a" exitCode=0 Oct 31 07:51:04.169880 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:04.169815 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerDied","Data":"02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a"} Oct 31 07:51:05.175566 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.175530 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerStarted","Data":"d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b"} Oct 31 07:51:05.176048 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.175597 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerStarted","Data":"2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7"} Oct 31 07:51:05.176048 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.175918 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:51:05.176048 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.175980 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:51:05.177522 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.177487 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:05.177859 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.177841 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerStarted","Data":"1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f"} Oct 31 07:51:05.177909 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.177882 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerStarted","Data":"33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da"} Oct 31 07:51:05.178118 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.178101 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:51:05.178153 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.178147 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:51:05.179172 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.179147 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:51:05.231968 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.231921 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podStartSLOduration=2.417847718 podStartE2EDuration="6.231908595s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.814060877s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:05.229231967 +0000 UTC m=+1309.701626215" watchObservedRunningTime="2026-10-31 07:51:05.231908595 +0000 UTC m=+1309.704302843" Oct 31 07:51:05.271998 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:05.271958 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podStartSLOduration=3.526711076 podStartE2EDuration="7.271945995s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.745234919s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:51:05.271289816 +0000 UTC m=+1309.743684067" watchObservedRunningTime="2026-10-31 07:51:05.271945995 +0000 UTC m=+1309.744340240" Oct 31 07:51:06.095530 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:51:06.095496 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:51:06.181582 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:06.181528 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:51:06.182033 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:06.181609 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:08.796802 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:08.796762 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:51:11.186608 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:11.186572 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:51:11.187171 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:11.186997 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:51:11.187171 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:11.187024 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:11.187407 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:11.187386 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:51:19.980087 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:19.980047 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:51:20.083159 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:20.083129 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:51:21.187519 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:21.187484 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:21.187927 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:21.187484 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:51:21.377432 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:21.377400 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:51:26.871624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:26.871591 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:51:26.976546 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:26.976510 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:51:31.188053 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:31.188017 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:31.188528 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:31.188013 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:51:36.096310 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:51:36.096269 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:51:41.187998 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:41.187961 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:51:41.188391 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:41.187964 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:51.188030 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:51.187985 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:51:51.188565 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:51:51.187986 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:52:01.187616 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:01.187579 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:52:01.188125 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:01.187586 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:52:06.095030 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:52:06.094993 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:52:08.913191 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:08.913147 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:52:11.187948 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:11.187906 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:52:11.188332 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:11.187908 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.42:8080: connect: connection refused" Oct 31 07:52:21.187734 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:21.187693 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused" Oct 31 07:52:21.188245 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:21.187941 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:52:31.188056 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:31.187981 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:52:34.076624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:34.076592 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5"] Oct 31 07:52:34.077087 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:34.076865 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" containerID="cri-o://2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7" gracePeriod=30 Oct 31 07:52:34.077087 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:34.076961 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kube-rbac-proxy" containerID="cri-o://d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b" gracePeriod=30 Oct 31 07:52:34.152999 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:34.152961 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:52:34.548680 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:34.548645 4870 generic.go:356] "Generic (PLEG): container finished" podID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerID="d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b" exitCode=0 Oct 31 07:52:34.548869 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:34.548706 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerDied","Data":"d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b"} Oct 31 07:52:36.095701 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:52:36.095658 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:52:36.182232 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:36.182186 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.42:8643/healthz\": dial tcp 10.132.0.42:8643: connect: connection refused" Oct 31 07:52:38.823244 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.823219 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:52:38.938504 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938429 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls\" (UniqueName: \"kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls\") pod \"554e5562-bed4-4c20-aff8-1d2e2ec57462\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " Oct 31 07:52:38.938504 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938468 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert\") pod \"554e5562-bed4-4c20-aff8-1d2e2ec57462\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " Oct 31 07:52:38.938717 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938527 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location\") pod \"554e5562-bed4-4c20-aff8-1d2e2ec57462\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " Oct 31 07:52:38.938717 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938567 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h\" (UniqueName: \"kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h\") pod \"554e5562-bed4-4c20-aff8-1d2e2ec57462\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " Oct 31 07:52:38.938717 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938621 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config\") pod \"554e5562-bed4-4c20-aff8-1d2e2ec57462\" (UID: \"554e5562-bed4-4c20-aff8-1d2e2ec57462\") " Oct 31 07:52:38.938946 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938913 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location" pod "554e5562-bed4-4c20-aff8-1d2e2ec57462" (UID: "554e5562-bed4-4c20-aff8-1d2e2ec57462"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:52:38.939030 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.938956 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert" pod "554e5562-bed4-4c20-aff8-1d2e2ec57462" (UID: "554e5562-bed4-4c20-aff8-1d2e2ec57462"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:52:38.939030 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.939004 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config" pod "554e5562-bed4-4c20-aff8-1d2e2ec57462" (UID: "554e5562-bed4-4c20-aff8-1d2e2ec57462"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:52:38.940690 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.940666 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls" pod "554e5562-bed4-4c20-aff8-1d2e2ec57462" (UID: "554e5562-bed4-4c20-aff8-1d2e2ec57462"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:52:38.940772 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:38.940739 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h" pod "554e5562-bed4-4c20-aff8-1d2e2ec57462" (UID: "554e5562-bed4-4c20-aff8-1d2e2ec57462"). InnerVolumeSpecName "kube-api-access-xcq6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:52:39.039217 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.039186 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/554e5562-bed4-4c20-aff8-1d2e2ec57462-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:52:39.039217 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.039212 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:52:39.039384 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.039222 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/554e5562-bed4-4c20-aff8-1d2e2ec57462-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:52:39.039384 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.039237 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcq6h\" (UniqueName: \"kubernetes.io/projected/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-api-access-xcq6h\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:52:39.039384 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.039246 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/554e5562-bed4-4c20-aff8-1d2e2ec57462-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:52:39.570354 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.570322 4870 generic.go:356] "Generic (PLEG): container finished" podID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerID="2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7" exitCode=0 Oct 31 07:52:39.570521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.570363 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerDied","Data":"2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7"} Oct 31 07:52:39.570521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.570378 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" Oct 31 07:52:39.570521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.570412 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5" event={"ID":"554e5562-bed4-4c20-aff8-1d2e2ec57462","Type":"ContainerDied","Data":"bea6f411cdb106c100fb305f4f1eef524268a8731e3a1a2b6d8c9fc73669a394"} Oct 31 07:52:39.570521 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.570438 4870 scope.go:122] "RemoveContainer" containerID="d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b" Oct 31 07:52:39.579068 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.579050 4870 scope.go:122] "RemoveContainer" containerID="2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7" Oct 31 07:52:39.586714 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.586695 4870 scope.go:122] "RemoveContainer" containerID="02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a" Oct 31 07:52:39.593691 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.593671 4870 scope.go:122] "RemoveContainer" containerID="d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b" Oct 31 07:52:39.593955 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:52:39.593931 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b\": container with ID starting with d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b not found: ID does not exist" containerID="d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b" Oct 31 07:52:39.594034 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.593969 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b"} err="failed to get container status \"d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b\": rpc error: code = NotFound desc = could not find container \"d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b\": container with ID starting with d403d04f8ee8459eba121c68e0ed4871520d6b8f3df09fcb1a51baf9bfbe252b not found: ID does not exist" Oct 31 07:52:39.594034 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.593997 4870 scope.go:122] "RemoveContainer" containerID="2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7" Oct 31 07:52:39.594230 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:52:39.594211 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7\": container with ID starting with 2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7 not found: ID does not exist" containerID="2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7" Oct 31 07:52:39.594265 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.594239 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7"} err="failed to get container status \"2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7\": rpc error: code = NotFound desc = could not find container \"2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7\": container with ID starting with 2dfc37d0445c0e41d8a5f9f2badacc366bd13e018313794b8338b5d6241c6eb7 not found: ID does not exist" Oct 31 07:52:39.594265 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.594254 4870 scope.go:122] "RemoveContainer" containerID="02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a" Oct 31 07:52:39.594453 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:52:39.594435 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a\": container with ID starting with 02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a not found: ID does not exist" containerID="02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a" Oct 31 07:52:39.594505 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.594461 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a"} err="failed to get container status \"02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a\": rpc error: code = NotFound desc = could not find container \"02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a\": container with ID starting with 02c95584c16558fb54d11779bdbe7a55913ed0832eb988251af093b278a58e6a not found: ID does not exist" Oct 31 07:52:39.633475 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.633443 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5"] Oct 31 07:52:39.644384 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.644362 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-predictor-574fc57cbd-r2ck5"] Oct 31 07:52:39.661552 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:39.661530 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" path="/var/lib/kubelet/pods/554e5562-bed4-4c20-aff8-1d2e2ec57462/volumes" Oct 31 07:52:40.097016 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:52:40.096983 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:53:06.097111 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:53:06.097074 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:53:10.782059 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:53:10.782028 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:53:36.097950 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:53:36.097915 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:54:04.716018 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:04.715942 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2"] Oct 31 07:54:04.718566 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:04.716275 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" containerID="cri-o://33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da" gracePeriod=30 Oct 31 07:54:04.718566 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:04.716309 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kube-rbac-proxy" containerID="cri-o://1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f" gracePeriod=30 Oct 31 07:54:04.767925 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:04.767895 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:04.915873 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:04.915838 4870 generic.go:356] "Generic (PLEG): container finished" podID="6958588f-c618-48c0-9315-954e82b8561c" containerID="1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f" exitCode=0 Oct 31 07:54:04.916020 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:04.915892 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerDied","Data":"1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f"} Oct 31 07:54:06.099244 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:54:06.099210 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:54:06.182786 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:06.182751 4870 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused" Oct 31 07:54:09.144964 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.144931 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:09.570548 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.570519 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:54:09.670578 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.670548 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location\") pod \"6958588f-c618-48c0-9315-954e82b8561c\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " Oct 31 07:54:09.670742 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.670591 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config\") pod \"6958588f-c618-48c0-9315-954e82b8561c\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " Oct 31 07:54:09.670742 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.670617 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8\" (UniqueName: \"kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8\") pod \"6958588f-c618-48c0-9315-954e82b8561c\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " Oct 31 07:54:09.670742 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.670645 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls\") pod \"6958588f-c618-48c0-9315-954e82b8561c\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " Oct 31 07:54:09.670742 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.670665 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert\") pod \"6958588f-c618-48c0-9315-954e82b8561c\" (UID: \"6958588f-c618-48c0-9315-954e82b8561c\") " Oct 31 07:54:09.670980 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.670899 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location" pod "6958588f-c618-48c0-9315-954e82b8561c" (UID: "6958588f-c618-48c0-9315-954e82b8561c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 07:54:09.671061 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.671034 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config" pod "6958588f-c618-48c0-9315-954e82b8561c" (UID: "6958588f-c618-48c0-9315-954e82b8561c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:54:09.671151 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.671128 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert" pod "6958588f-c618-48c0-9315-954e82b8561c" (UID: "6958588f-c618-48c0-9315-954e82b8561c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:54:09.672717 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.672698 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8" pod "6958588f-c618-48c0-9315-954e82b8561c" (UID: "6958588f-c618-48c0-9315-954e82b8561c"). InnerVolumeSpecName "kube-api-access-cf7z8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:54:09.672834 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.672736 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls" pod "6958588f-c618-48c0-9315-954e82b8561c" (UID: "6958588f-c618-48c0-9315-954e82b8561c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:54:09.771383 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.771323 4870 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6958588f-c618-48c0-9315-954e82b8561c-kserve-provision-location\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:54:09.771383 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.771347 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:54:09.771383 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.771356 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf7z8\" (UniqueName: \"kubernetes.io/projected/6958588f-c618-48c0-9315-954e82b8561c-kube-api-access-cf7z8\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:54:09.771383 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.771365 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6958588f-c618-48c0-9315-954e82b8561c-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:54:09.771383 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.771374 4870 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6958588f-c618-48c0-9315-954e82b8561c-cabundle-cert\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:54:09.936852 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.936819 4870 generic.go:356] "Generic (PLEG): container finished" podID="6958588f-c618-48c0-9315-954e82b8561c" containerID="33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da" exitCode=0 Oct 31 07:54:09.937089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.936896 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" Oct 31 07:54:09.937089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.936905 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerDied","Data":"33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da"} Oct 31 07:54:09.937089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.936969 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2" event={"ID":"6958588f-c618-48c0-9315-954e82b8561c","Type":"ContainerDied","Data":"ffc2b06eb014e44495c5feb77fc42cce2d79e8e7fa79e9c1a0210d182c5a1c6e"} Oct 31 07:54:09.937089 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.936996 4870 scope.go:122] "RemoveContainer" containerID="1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f" Oct 31 07:54:09.946148 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.946131 4870 scope.go:122] "RemoveContainer" containerID="33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da" Oct 31 07:54:09.953701 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.953685 4870 scope.go:122] "RemoveContainer" containerID="918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332" Oct 31 07:54:09.960732 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.960716 4870 scope.go:122] "RemoveContainer" containerID="1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f" Oct 31 07:54:09.961000 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:54:09.960981 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f\": container with ID starting with 1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f not found: ID does not exist" containerID="1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f" Oct 31 07:54:09.961080 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.961012 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f"} err="failed to get container status \"1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f\": rpc error: code = NotFound desc = could not find container \"1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f\": container with ID starting with 1e5f81ab9ac5f13735b267e610b882636fed8721953e1dc55c0eae51d241243f not found: ID does not exist" Oct 31 07:54:09.961080 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.961035 4870 scope.go:122] "RemoveContainer" containerID="33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da" Oct 31 07:54:09.961323 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:54:09.961303 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da\": container with ID starting with 33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da not found: ID does not exist" containerID="33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da" Oct 31 07:54:09.961367 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.961330 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da"} err="failed to get container status \"33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da\": rpc error: code = NotFound desc = could not find container \"33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da\": container with ID starting with 33db0763bd26a4fc80d66abf0ef55006d2bc4a9fc63f3711aafc58e3cc1271da not found: ID does not exist" Oct 31 07:54:09.961367 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.961346 4870 scope.go:122] "RemoveContainer" containerID="918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332" Oct 31 07:54:09.961585 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:54:09.961567 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332\": container with ID starting with 918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332 not found: ID does not exist" containerID="918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332" Oct 31 07:54:09.961625 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.961591 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332"} err="failed to get container status \"918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332\": rpc error: code = NotFound desc = could not find container \"918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332\": container with ID starting with 918364e21228a21f7e4a506ce53d6514fd83d07eb4909e4706c616192d4a0332 not found: ID does not exist" Oct 31 07:54:09.986721 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.986698 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2"] Oct 31 07:54:09.998453 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:09.998433 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-7281997251/isvc-canary-promote-v2-predictor-6bc9f6df57-nmsb2"] Oct 31 07:54:11.661837 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:11.661808 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6958588f-c618-48c0-9315-954e82b8561c" path="/var/lib/kubelet/pods/6958588f-c618-48c0-9315-954e82b8561c/volumes" Oct 31 07:54:11.693490 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:11.693465 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:15.789643 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:15.789615 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:54:15.796708 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:15.796681 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:54:18.383851 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:18.383782 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:18.485396 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:18.485365 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:21.259357 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:21.259322 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:21.362780 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:21.362753 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:22.584082 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.584047 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:22.735663 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.735597 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:22.772098 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772065 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4"] Oct 31 07:54:22.772429 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772414 4870 cpu_manager.go:414] "containerMap: removing container" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="storage-initializer" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772432 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="storage-initializer" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772442 4870 cpu_manager.go:414] "containerMap: removing container" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kube-rbac-proxy" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772448 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kube-rbac-proxy" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772458 4870 cpu_manager.go:414] "containerMap: removing container" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="storage-initializer" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772463 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="storage-initializer" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772471 4870 cpu_manager.go:414] "containerMap: removing container" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772476 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772489 4870 cpu_manager.go:414] "containerMap: removing container" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" Oct 31 07:54:22.772494 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772494 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" Oct 31 07:54:22.772826 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772504 4870 cpu_manager.go:414] "containerMap: removing container" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kube-rbac-proxy" Oct 31 07:54:22.772826 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772509 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kube-rbac-proxy" Oct 31 07:54:22.772826 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772563 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kserve-container" Oct 31 07:54:22.772826 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772572 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kube-rbac-proxy" Oct 31 07:54:22.772826 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772581 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="6958588f-c618-48c0-9315-954e82b8561c" containerName="kube-rbac-proxy" Oct 31 07:54:22.772826 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.772588 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="554e5562-bed4-4c20-aff8-1d2e2ec57462" containerName="kserve-container" Oct 31 07:54:22.777072 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.777054 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.794187 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.794159 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-166168988c\"/\"message-dumper-raw-cipn-predictor-serving-cert\"" Oct 31 07:54:22.806398 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.806369 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-166168988c\"/\"openshift-service-ca.crt\"" Oct 31 07:54:22.806398 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.806379 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-166168988c\"/\"kube-root-ca.crt\"" Oct 31 07:54:22.806584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.806408 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-166168988c\"/\"default-dockercfg-s48kc\"" Oct 31 07:54:22.806584 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.806408 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-166168988c\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\"" Oct 31 07:54:22.807497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.807475 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4"] Oct 31 07:54:22.862242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.862159 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9g2\" (UniqueName: \"kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.862242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.862196 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.862242 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.862222 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.963209 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.963172 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.963371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.963247 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9g2\" (UniqueName: \"kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.963371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.963273 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.964001 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.963975 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.965601 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.965575 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:22.986286 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:22.986252 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9g2\" (UniqueName: \"kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2\") pod \"message-dumper-raw-cipn-predictor-5949489dd-5g5j4\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:23.091906 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:23.091869 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:23.387317 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:23.387176 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4"] Oct 31 07:54:23.390176 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:54:23.390114 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6cde42a_2b5b_47d7_aab7_886bf6e2a285.slice/crio-dcf3077c9d4131a5cc45dd04113058a32faf9b9ffecb1c8a50db1d79b648e3d2 WatchSource:0}: Error finding container dcf3077c9d4131a5cc45dd04113058a32faf9b9ffecb1c8a50db1d79b648e3d2: Status 404 returned error can't find the container with id dcf3077c9d4131a5cc45dd04113058a32faf9b9ffecb1c8a50db1d79b648e3d2 Oct 31 07:54:23.392184 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:23.392164 4870 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 07:54:23.996153 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:23.996116 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" event={"ID":"e6cde42a-2b5b-47d7-aab7-886bf6e2a285","Type":"ContainerStarted","Data":"dcf3077c9d4131a5cc45dd04113058a32faf9b9ffecb1c8a50db1d79b648e3d2"} Oct 31 07:54:25.002104 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:25.002020 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" event={"ID":"e6cde42a-2b5b-47d7-aab7-886bf6e2a285","Type":"ContainerStarted","Data":"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28"} Oct 31 07:54:25.002104 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:25.002087 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" event={"ID":"e6cde42a-2b5b-47d7-aab7-886bf6e2a285","Type":"ContainerStarted","Data":"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b"} Oct 31 07:54:25.002474 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:25.002128 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:25.043274 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:25.043225 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" podStartSLOduration=1.690057014 podStartE2EDuration="3.043212135s" totalImagesPullingTime="1.353155121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 07:54:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:54:25.042813473 +0000 UTC m=+1509.515207720" watchObservedRunningTime="2026-10-31 07:54:25.043212135 +0000 UTC m=+1509.515606383" Oct 31 07:54:26.005875 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:26.005843 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:26.007558 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:26.007533 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:33.018254 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:33.018222 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:54:36.097141 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:54:36.097108 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:54:36.724102 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:36.724074 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:42.778617 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:42.778586 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:54:42.906105 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:54:42.906075 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:55:06.096708 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:55:06.096661 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:55:09.288277 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:55:09.288241 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:55:36.095698 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:55:36.095611 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:55:56.937561 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:55:56.937517 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:06.095941 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:56:06.095905 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:56:09.367440 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:09.367410 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:18.152537 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:18.152507 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-166168988c_message-dumper-raw-cipn-predictor-5949489dd-5g5j4_e6cde42a-2b5b-47d7-aab7-886bf6e2a285/kserve-container/0.log" Oct 31 07:56:18.781604 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:18.781572 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:18.853641 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:18.853611 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4"] Oct 31 07:56:18.853899 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:18.853872 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kserve-container" containerID="cri-o://648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b" gracePeriod=30 Oct 31 07:56:18.853985 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:18.853878 4870 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kube-rbac-proxy" containerID="cri-o://8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28" gracePeriod=30 Oct 31 07:56:18.897683 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:18.897647 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:19.102870 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.102844 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:56:19.124371 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.124345 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2\" (UniqueName: \"kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2\") pod \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " Oct 31 07:56:19.124497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.124389 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config\") pod \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " Oct 31 07:56:19.124497 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.124448 4870 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls\") pod \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\" (UID: \"e6cde42a-2b5b-47d7-aab7-886bf6e2a285\") " Oct 31 07:56:19.124772 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.124750 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config" pod "e6cde42a-2b5b-47d7-aab7-886bf6e2a285" (UID: "e6cde42a-2b5b-47d7-aab7-886bf6e2a285"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 07:56:19.126455 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.126431 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls" pod "e6cde42a-2b5b-47d7-aab7-886bf6e2a285" (UID: "e6cde42a-2b5b-47d7-aab7-886bf6e2a285"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 07:56:19.126554 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.126433 4870 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2" pod "e6cde42a-2b5b-47d7-aab7-886bf6e2a285" (UID: "e6cde42a-2b5b-47d7-aab7-886bf6e2a285"). InnerVolumeSpecName "kube-api-access-sz9g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 07:56:19.225358 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.225325 4870 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz9g2\" (UniqueName: \"kubernetes.io/projected/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-api-access-sz9g2\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:56:19.225358 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.225352 4870 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:56:19.225754 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.225365 4870 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e6cde42a-2b5b-47d7-aab7-886bf6e2a285-proxy-tls\") on node \"ip-10-0-140-6.ec2.internal\" DevicePath \"\"" Oct 31 07:56:19.455054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.454970 4870 generic.go:356] "Generic (PLEG): container finished" podID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerID="8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28" exitCode=0 Oct 31 07:56:19.455054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.455001 4870 generic.go:356] "Generic (PLEG): container finished" podID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerID="648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b" exitCode=2 Oct 31 07:56:19.455054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.455022 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" event={"ID":"e6cde42a-2b5b-47d7-aab7-886bf6e2a285","Type":"ContainerDied","Data":"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28"} Oct 31 07:56:19.455054 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.455039 4870 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" Oct 31 07:56:19.455340 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.455075 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" event={"ID":"e6cde42a-2b5b-47d7-aab7-886bf6e2a285","Type":"ContainerDied","Data":"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b"} Oct 31 07:56:19.455340 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.455097 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4" event={"ID":"e6cde42a-2b5b-47d7-aab7-886bf6e2a285","Type":"ContainerDied","Data":"dcf3077c9d4131a5cc45dd04113058a32faf9b9ffecb1c8a50db1d79b648e3d2"} Oct 31 07:56:19.455340 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.455119 4870 scope.go:122] "RemoveContainer" containerID="8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28" Oct 31 07:56:19.463278 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.463262 4870 scope.go:122] "RemoveContainer" containerID="648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b" Oct 31 07:56:19.470605 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.470588 4870 scope.go:122] "RemoveContainer" containerID="8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28" Oct 31 07:56:19.470872 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:56:19.470853 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28\": container with ID starting with 8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28 not found: ID does not exist" containerID="8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28" Oct 31 07:56:19.470926 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.470881 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28"} err="failed to get container status \"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28\": rpc error: code = NotFound desc = could not find container \"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28\": container with ID starting with 8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28 not found: ID does not exist" Oct 31 07:56:19.470926 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.470899 4870 scope.go:122] "RemoveContainer" containerID="648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b" Oct 31 07:56:19.471122 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:56:19.471104 4870 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b\": container with ID starting with 648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b not found: ID does not exist" containerID="648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b" Oct 31 07:56:19.471163 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.471130 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b"} err="failed to get container status \"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b\": rpc error: code = NotFound desc = could not find container \"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b\": container with ID starting with 648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b not found: ID does not exist" Oct 31 07:56:19.471163 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.471145 4870 scope.go:122] "RemoveContainer" containerID="8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28" Oct 31 07:56:19.471330 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.471314 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28"} err="failed to get container status \"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28\": rpc error: code = NotFound desc = could not find container \"8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28\": container with ID starting with 8a01f8d005c2e311ce8f386eebd718c8087aae7c26aa13c037fef44b0a206c28 not found: ID does not exist" Oct 31 07:56:19.471374 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.471329 4870 scope.go:122] "RemoveContainer" containerID="648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b" Oct 31 07:56:19.471514 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.471498 4870 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b"} err="failed to get container status \"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b\": rpc error: code = NotFound desc = could not find container \"648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b\": container with ID starting with 648fbe2ba6bc7a43a1927dd75b7bd989acae11049eca9dfbf1141e9f01669f0b not found: ID does not exist" Oct 31 07:56:19.502940 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.502920 4870 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4"] Oct 31 07:56:19.514684 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.514661 4870 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-166168988c/message-dumper-raw-cipn-predictor-5949489dd-5g5j4"] Oct 31 07:56:19.661955 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:19.661917 4870 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" path="/var/lib/kubelet/pods/e6cde42a-2b5b-47d7-aab7-886bf6e2a285/volumes" Oct 31 07:56:36.096357 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:56:36.096318 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:56:50.122322 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:50.122290 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:51.767729 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:51.767666 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:51.874043 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:51.874005 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:55.891047 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:55.891016 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:56:55.992886 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:56:55.992851 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:06.096319 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:57:06.096237 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:57:09.506599 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:09.506539 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:28.638190 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:28.638153 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:28.940737 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:28.940654 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:29.044252 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:29.044220 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:33.298409 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:33.298372 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-l5c85_2c31ebf7-17d4-4aef-824b-afd78bd52d6e/global-pull-secret-syncer/0.log" Oct 31 07:57:33.707871 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:33.707775 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6sww9_33d0e8e1-a3d4-4a15-a9bc-17cde0e5df6e/connection-checker/0.log" Oct 31 07:57:33.832425 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:33.832396 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-v5stm_b67627fb-3093-4fff-9ab1-c0a4432f72e2/connection-checker/0.log" Oct 31 07:57:33.956527 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:33.956500 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-xzc9s_35dff775-deae-46eb-a3bc-b5618173818f/connection-checker/0.log" Oct 31 07:57:34.090931 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:34.090900 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" Oct 31 07:57:34.131732 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:34.131699 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:34.234454 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:34.234421 4870 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2qrrp_eb684428-62fd-4dc0-aa16-3ba4dfc7c89b/konnectivity-agent/0.log" limit=1024 Oct 31 07:57:34.690926 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:34.690890 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-6.ec2.internal_8952dcaf440f7a0e696c251b7a42e952/haproxy/0.log" Oct 31 07:57:36.095591 ip-10-0-140-6 kubenswrapper[4870]: E1031 07:57:36.095557 4870 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-zdg8b" containerName="extractor" Oct 31 07:57:40.616781 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:40.616748 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-vq7hc_92342503-edc9-45e9-b9d7-fff71ce5092d/cluster-monitoring-operator/0.log" Oct 31 07:57:42.079085 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.079047 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vxgj7_31e2536a-e888-4f15-9d91-52e6cd67aae2/node-exporter/0.log" Oct 31 07:57:42.201168 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.201139 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vxgj7_31e2536a-e888-4f15-9d91-52e6cd67aae2/kube-rbac-proxy/0.log" Oct 31 07:57:42.325691 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.325662 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vxgj7_31e2536a-e888-4f15-9d91-52e6cd67aae2/init-textfile/0.log" Oct 31 07:57:42.517378 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517298 4870 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj"] Oct 31 07:57:42.517621 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517608 4870 cpu_manager.go:414] "containerMap: removing container" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kube-rbac-proxy" Oct 31 07:57:42.517674 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517623 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kube-rbac-proxy" Oct 31 07:57:42.517674 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517641 4870 cpu_manager.go:414] "containerMap: removing container" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kserve-container" Oct 31 07:57:42.517674 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517646 4870 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kserve-container" Oct 31 07:57:42.517782 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517722 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kube-rbac-proxy" Oct 31 07:57:42.517782 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.517732 4870 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6cde42a-2b5b-47d7-aab7-886bf6e2a285" containerName="kserve-container" Oct 31 07:57:42.520647 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.520629 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.537280 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.537252 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kj9n5\"/\"openshift-service-ca.crt\"" Oct 31 07:57:42.538991 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.538970 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj"] Oct 31 07:57:42.560940 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.560914 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25k4\" (UniqueName: \"kubernetes.io/projected/d9378e6b-cb60-4a65-ae97-d2381740e051-kube-api-access-z25k4\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.561061 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.560946 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-proc\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.561061 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.560969 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-lib-modules\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.561179 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.561124 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-sys\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.561179 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.561158 4870 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-podres\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.633523 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.633499 4870 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kj9n5\"/\"kube-root-ca.crt\"" Oct 31 07:57:42.633697 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.633545 4870 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kj9n5\"/\"default-dockercfg-pcw5m\"" Oct 31 07:57:42.662129 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662100 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-lib-modules\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662148 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-sys\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662171 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-podres\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662212 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25k4\" (UniqueName: \"kubernetes.io/projected/d9378e6b-cb60-4a65-ae97-d2381740e051-kube-api-access-z25k4\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662232 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-sys\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662239 4870 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-proc\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662261 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662234 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-lib-modules\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662451 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662281 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-podres\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.662451 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.662309 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d9378e6b-cb60-4a65-ae97-d2381740e051-proc\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.684570 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.684542 4870 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25k4\" (UniqueName: \"kubernetes.io/projected/d9378e6b-cb60-4a65-ae97-d2381740e051-kube-api-access-z25k4\") pod \"perf-node-gather-daemonset-8bpcj\" (UID: \"d9378e6b-cb60-4a65-ae97-d2381740e051\") " pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:42.833311 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:42.833283 4870 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:43.124199 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:43.124175 4870 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj"] Oct 31 07:57:43.126684 ip-10-0-140-6 kubenswrapper[4870]: W1031 07:57:43.126654 4870 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd9378e6b_cb60_4a65_ae97_d2381740e051.slice/crio-66cd89a8428066235ab3e3d454337ef9f3477adeed3a1c3f3507d74d33c455a2 WatchSource:0}: Error finding container 66cd89a8428066235ab3e3d454337ef9f3477adeed3a1c3f3507d74d33c455a2: Status 404 returned error can't find the container with id 66cd89a8428066235ab3e3d454337ef9f3477adeed3a1c3f3507d74d33c455a2 Oct 31 07:57:43.780624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:43.780588 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" event={"ID":"d9378e6b-cb60-4a65-ae97-d2381740e051","Type":"ContainerStarted","Data":"4dbaecad6b844c0e8e16c300d9c70a0b480dcada73f540fa5e799b5a824b89cd"} Oct 31 07:57:43.780831 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:43.780644 4870 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" event={"ID":"d9378e6b-cb60-4a65-ae97-d2381740e051","Type":"ContainerStarted","Data":"66cd89a8428066235ab3e3d454337ef9f3477adeed3a1c3f3507d74d33c455a2"} Oct 31 07:57:43.780831 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:43.780727 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:46.318007 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:46.317979 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-p9pnl_5fec1a19-aa94-423d-a573-71c8b0739f55/networking-console-plugin/0.log" Oct 31 07:57:47.321536 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:47.321504 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/2.log" Oct 31 07:57:47.324699 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:47.324678 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-hzng6_a4a619c0-1d45-4bfe-b948-f79f3d461cb2/console-operator/3.log" Oct 31 07:57:49.378849 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:49.378779 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-cqqrp_5abb2415-7417-4478-9d12-a4d9f2e0549c/volume-data-source-validator/0.log" Oct 31 07:57:49.794751 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:49.794720 4870 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" Oct 31 07:57:49.834609 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:49.834559 4870 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-kj9n5/perf-node-gather-daemonset-8bpcj" podStartSLOduration=7.834545596 podStartE2EDuration="7.834545596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 07:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 07:57:43.822644612 +0000 UTC m=+1708.295038861" watchObservedRunningTime="2026-10-31 07:57:49.834545596 +0000 UTC m=+1714.306939843" Oct 31 07:57:51.439398 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:51.439370 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m99x4_8039f97e-ddca-4960-80a9-b53c24950ca9/dns/0.log" Oct 31 07:57:51.562500 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:51.562471 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m99x4_8039f97e-ddca-4960-80a9-b53c24950ca9/kube-rbac-proxy/0.log" Oct 31 07:57:52.406591 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:52.406559 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-sv7zk_a122982a-c398-47f1-a368-fd89db8899a7/dns-node-resolver/0.log" Oct 31 07:57:53.915170 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:53.915142 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m8msg_9e12a075-b4ec-44e3-8811-7b19f1410513/node-ca/0.log" Oct 31 07:57:55.738478 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:55.738448 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b4c78c6d5-pmxm4_af42539c-c267-4894-a291-764cbb70c30f/router/0.log" Oct 31 07:57:56.646624 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:56.646580 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-g29bz_e71bf711-37d2-421a-8bb5-a838f60d36dc/serve-healthcheck-canary/0.log" Oct 31 07:57:57.782444 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:57.782409 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-vvm6c_8f52e908-ef31-4672-ac4d-1ee9778d1b52/insights-operator/0.log" Oct 31 07:57:57.782876 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:57.782487 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-vvm6c_8f52e908-ef31-4672-ac4d-1ee9778d1b52/insights-operator/1.log" Oct 31 07:57:58.256950 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:58.256923 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdg8b_0e2c4370-aeba-4e68-ad8d-396664e91a83/kube-rbac-proxy/0.log" Oct 31 07:57:58.379384 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:58.379358 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdg8b_0e2c4370-aeba-4e68-ad8d-396664e91a83/exporter/0.log" Oct 31 07:57:58.501960 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:58.501926 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zdg8b_0e2c4370-aeba-4e68-ad8d-396664e91a83/extractor/0.log" Oct 31 07:57:59.095927 ip-10-0-140-6 kubenswrapper[4870]: I1031 07:57:59.095899 4870 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-z54zk-w4l76_143aebad-6cd8-46bd-ac09-af425fe81be7/insights-gathering/0.log"