Oct 30 02:55:12.662563 ip-10-0-142-236 systemd[1]: Starting Kubernetes Kubelet... Oct 30 02:55:13.033287 ip-10-0-142-236 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:55:13.033287 ip-10-0-142-236 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 02:55:13.033287 ip-10-0-142-236 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:55:13.033287 ip-10-0-142-236 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:55:13.037155 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037074 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.037155 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037154 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.037155 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037158 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037161 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037164 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037167 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037170 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037173 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037176 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037179 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037182 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037185 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037187 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037190 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037192 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037195 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037197 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037200 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037203 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037205 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037208 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.037263 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037210 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037219 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037223 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037227 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037230 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037233 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037236 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037239 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037242 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037244 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037261 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037266 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037271 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037275 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037280 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037284 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037289 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037292 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.037705 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037294 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037297 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037300 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037303 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037306 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037309 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037312 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037315 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037317 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037320 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037322 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037325 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037327 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037330 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037333 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037336 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037339 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037342 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037344 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.038179 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037347 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037349 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037352 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037355 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037357 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037360 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037364 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037368 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037371 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037373 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037376 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037378 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037381 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037384 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037386 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037389 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037392 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037395 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037397 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.038640 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037401 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037404 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037406 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037409 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037411 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037414 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037417 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037419 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037424 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037437 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037440 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037443 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037445 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037448 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037451 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037453 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037456 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037459 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037462 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037472 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.039119 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037475 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037478 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037482 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037486 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037489 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037491 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037494 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.037497 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038322 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038331 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038334 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038337 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038340 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038344 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038347 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038351 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038353 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038356 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038359 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038361 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.039589 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038364 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038367 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038370 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038372 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038375 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038378 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038380 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038383 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038386 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038389 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038392 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038395 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038397 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038400 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038402 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038406 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038410 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038413 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038417 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.040071 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038420 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038422 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038426 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038429 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038432 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038434 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038437 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038440 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038443 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038446 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038449 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038451 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038454 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038456 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038459 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038462 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038465 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038467 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038470 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038472 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.040545 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038475 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038478 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038482 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038485 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038488 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038490 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038493 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038496 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038498 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038501 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038503 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038506 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038509 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038511 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038515 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038517 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038520 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038522 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038525 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.041031 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038528 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038531 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038534 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038537 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038539 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038542 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038546 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038549 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038551 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038554 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038556 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038559 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038562 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038565 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038567 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038570 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038573 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038575 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038578 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.041490 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038580 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038583 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038585 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038588 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038590 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038593 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038595 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038598 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038602 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038614 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038617 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038622 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038625 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038628 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038630 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038634 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.038638 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039233 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039241 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039244 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.041944 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039260 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039264 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039266 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039269 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039272 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039274 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039277 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039280 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039282 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039285 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039287 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039291 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039293 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039296 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039298 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039301 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039304 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039306 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039309 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039312 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.042418 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039317 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039321 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039324 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039329 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039332 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039334 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039337 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039346 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039349 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039352 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039354 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039357 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039362 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039365 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039367 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039371 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039374 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039376 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039379 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.042889 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039381 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039384 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039386 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039389 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039391 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039394 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039396 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039399 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039401 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039404 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039407 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039409 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039412 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039414 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039417 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039422 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039424 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039427 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039429 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039432 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.043384 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039440 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039443 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039445 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039448 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039451 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039453 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039456 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039458 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039461 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039463 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039466 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039468 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039471 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039478 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039481 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039483 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039486 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039488 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039491 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.043860 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039493 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039496 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039498 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039501 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039503 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039506 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039508 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039511 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039516 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039518 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039521 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039523 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039526 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039529 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039532 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039534 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039537 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039539 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039542 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.044344 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039546 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039551 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039554 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039556 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.039559 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040549 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040561 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040572 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040577 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040582 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040585 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040590 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040594 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040597 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040601 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040604 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040607 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040611 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040614 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040616 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040619 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040623 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040626 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 02:55:13.044789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040633 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040636 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040640 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040643 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040646 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040655 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040658 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040661 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040664 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040668 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040671 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040674 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040677 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040680 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040684 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040688 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040690 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040693 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040696 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040699 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040705 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040708 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040711 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040714 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 02:55:13.045421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040717 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040721 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040724 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040727 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040730 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040733 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040737 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040740 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040744 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040747 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040750 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040753 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040757 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040760 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040764 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040767 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040770 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040773 2578 flags.go:64] FLAG: --help="false" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040776 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040779 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040782 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040785 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040788 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 02:55:13.045998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040792 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040795 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040797 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040800 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040803 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040806 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040809 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040812 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040815 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040818 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040821 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040824 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040827 2578 flags.go:64] FLAG: --lock-file="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040830 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040833 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040836 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040842 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040847 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040850 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040853 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040856 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040859 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040862 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040865 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040870 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 02:55:13.046602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040873 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040877 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040880 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040883 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040886 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040889 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040892 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040895 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040898 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040905 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040908 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040911 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040914 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040917 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040920 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040923 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040926 2578 flags.go:64] FLAG: --port="10250" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040929 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040932 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09939771943171ea3" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040935 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040938 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040941 2578 flags.go:64] FLAG: --register-node="true" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040944 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040948 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040960 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040964 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 02:55:13.047228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040967 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040971 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040974 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040977 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040979 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040983 2578 flags.go:64] FLAG: --runonce="false" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040995 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.040998 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041001 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041004 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041007 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041010 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041013 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041016 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041019 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041022 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041025 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041028 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041031 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041034 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041036 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041042 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041045 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041050 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041054 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 02:55:13.047862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041057 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041060 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041063 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041066 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041069 2578 flags.go:64] FLAG: --v="2" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041074 2578 flags.go:64] FLAG: --version="false" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041079 2578 flags.go:64] FLAG: --vmodule="" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041083 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041086 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041236 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041239 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041243 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041246 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041263 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041267 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041271 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041276 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041279 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041282 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041284 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041287 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.048461 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041289 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041292 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041295 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041298 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041300 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041303 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041305 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041308 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041311 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041315 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041318 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041321 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041324 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041326 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041329 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041332 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041336 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041338 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041341 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.048948 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041344 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041346 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041349 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041351 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041354 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041357 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041359 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041362 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041365 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041368 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041370 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041373 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041376 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041380 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041384 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041387 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041389 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041393 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041395 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.049456 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041398 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041400 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041403 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041407 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041409 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041412 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041415 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041417 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041419 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041422 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041426 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041429 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041431 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041434 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041436 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041440 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041442 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041445 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041447 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041450 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.049906 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041452 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041456 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041458 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041461 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041463 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041466 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041469 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041471 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041473 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041476 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041478 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041481 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041483 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041486 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041488 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041492 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041496 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041500 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041503 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.050482 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041506 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041509 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041511 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041515 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041518 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041520 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041523 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041525 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041528 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041531 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041533 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041537 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041549 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041552 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041554 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041558 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.050940 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041560 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.041598 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041636 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041641 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041645 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041649 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041653 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041656 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041659 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041661 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041664 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041667 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041669 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041674 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041678 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.051332 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041680 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041683 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041686 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041689 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041692 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041695 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041698 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041700 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041703 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041705 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041708 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041711 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041713 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041716 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041719 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041721 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041724 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041727 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041729 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041732 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.051712 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041734 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041737 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041739 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041742 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041745 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041747 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041750 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041752 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041755 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041757 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041761 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041765 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041769 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041772 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041775 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041777 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041781 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041784 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041787 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.052199 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041789 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041792 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041794 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041797 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041800 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041802 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041805 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041807 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041812 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041814 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041817 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041820 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041823 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041826 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041828 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041831 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041849 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041853 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041856 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041859 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.052724 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041862 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041865 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041867 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041871 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041874 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041876 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041879 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041882 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041884 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041889 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041892 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041894 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041897 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041900 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041903 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041906 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041909 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041912 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041914 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.053205 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041917 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041919 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041922 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041924 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041927 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041930 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041933 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041935 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041938 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041940 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041943 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041945 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041948 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041951 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.041953 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.048563 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.048577 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.048591 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.048596 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 02:55:13.053806 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048654 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048659 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048663 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048666 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048668 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048671 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048674 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048677 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048682 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048685 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048688 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048690 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048694 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048697 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048700 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048703 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048705 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048709 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048711 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.054297 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048714 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048717 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048719 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048722 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048725 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048727 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048730 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048733 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048735 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048739 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048741 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048744 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048747 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048749 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048752 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048754 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048757 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048760 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048763 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048765 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.054846 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048768 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048770 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048773 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048776 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048778 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048781 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048783 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048786 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048788 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048791 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048793 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048796 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048799 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048802 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048804 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048807 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048809 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048811 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048814 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.055370 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048817 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048819 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048822 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048825 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048829 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048833 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048836 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048839 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048841 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048844 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048847 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048849 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048852 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048854 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048857 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048859 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048862 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048864 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048867 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.055862 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048869 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048872 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048874 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048877 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048879 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048882 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048885 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048887 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048890 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048894 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048897 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048900 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048902 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048905 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048907 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048911 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048914 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048916 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048919 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.056348 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048922 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048925 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048927 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048930 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048932 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048935 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048937 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048939 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.048942 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.048970 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049115 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049120 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049123 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049126 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049128 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.056794 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049131 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049133 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049136 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049140 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049143 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049147 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049150 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049153 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049155 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049158 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049161 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049163 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049166 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049168 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049172 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049175 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049178 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049181 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049183 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.057147 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049186 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049188 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049191 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049193 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049196 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049199 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049201 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049204 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049207 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049209 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049212 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049214 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049217 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049219 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049222 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049225 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049227 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049229 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049232 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049235 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.057784 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049238 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049241 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049243 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049246 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049263 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049266 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049268 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049271 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049274 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049276 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049279 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049281 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049284 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049286 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049289 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049291 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049294 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049296 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049299 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049301 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.058582 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049304 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049306 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049309 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049311 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049314 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049316 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049319 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049321 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049324 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049326 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049329 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049331 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049334 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049337 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049340 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049343 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049345 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049348 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049350 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049353 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.059224 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049355 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049358 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049360 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049363 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049365 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049368 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049370 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049372 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049375 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049377 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049380 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049382 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049385 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049387 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049390 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049392 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049395 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049397 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049400 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.059780 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049402 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049405 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049407 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.049435 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049471 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049476 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049481 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049484 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049487 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049490 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049492 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049497 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049500 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049503 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:55:13.060730 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049505 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049509 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049512 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049515 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049518 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049521 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049523 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049526 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049528 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049531 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049534 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049536 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049539 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049541 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049544 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049547 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049549 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049552 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049554 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049557 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:55:13.061112 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049559 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049563 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049566 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049568 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049571 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049574 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049580 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049583 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049585 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049588 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049591 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049594 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049596 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049599 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049602 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049605 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049607 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049610 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049612 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:55:13.061599 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049615 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049617 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049620 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049623 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049625 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049628 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049630 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049633 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049635 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049638 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049640 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049643 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049645 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049649 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049652 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049654 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049657 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049660 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049663 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049665 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:55:13.062092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049668 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049670 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049673 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049675 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049678 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049681 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049684 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049687 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049689 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049692 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049694 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049697 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049699 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049701 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049704 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049706 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049709 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049712 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049714 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:55:13.062699 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049716 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049719 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049721 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049724 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049726 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049729 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049732 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049735 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049738 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049740 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049743 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049747 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049750 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049753 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049755 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:13.049758 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.049808 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.051677 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 02:55:13.063213 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.052534 2578 server.go:1014] "Starting client certificate rotation" Oct 30 02:55:13.063719 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.052628 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:55:13.063719 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.053173 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:55:13.063719 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.054650 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:55:13.063719 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.057310 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:55:13.066602 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.066584 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 02:55:13.069753 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.069740 2578 log.go:25] "Validated CRI v1 image API" Oct 30 02:55:13.070654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.070628 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 02:55:13.073725 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.073701 2578 fs.go:135] Filesystem UUIDs: map[520720d4-3f43-488b-bc2c-8215a6cfaee1:/dev/nvme0n1p4 549e799a-b878-462d-8408-421e1b5b776e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 02:55:13.073776 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.073725 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 02:55:13.079793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.079673 2578 manager.go:217] Machine: {Timestamp:2026-10-30 02:55:13.078352108 +0000 UTC m=+0.310741011 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102118 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec5f04c246e0490ccaa23ac758501 SystemUUID:ec2ec5f0-4c24-6e04-90cc-aa23ac758501 BootID:287fe92e-ebd3-4185-80f0-6a709581f4e1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:cf:38:f5:c3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:cf:38:f5:c3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:aa:56:82:f3:64:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 02:55:13.079793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.079788 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 02:55:13.079910 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.079875 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 02:55:13.082335 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.082296 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 02:55:13.082507 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.082336 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 02:55:13.082559 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.082510 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 02:55:13.082559 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.082518 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 02:55:13.082559 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.082554 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:55:13.082637 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.082572 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:55:13.083322 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.083311 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 02:55:13.086871 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.086859 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 02:55:13.087007 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.086997 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 02:55:13.087697 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.087688 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 02:55:13.087727 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.087706 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 02:55:13.087727 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.087717 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 02:55:13.087727 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.087726 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 02:55:13.087830 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.087735 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 02:55:13.088098 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.088085 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:55:13.090322 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.090306 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 02:55:13.094517 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.094502 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:55:13.095265 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095233 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 02:55:13.095343 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095290 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 02:55:13.095512 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095497 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 02:55:13.095570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095527 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 02:55:13.095570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095537 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 02:55:13.095570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095546 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 02:55:13.095570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095554 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 02:55:13.095570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.095564 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 02:55:13.096067 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096057 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 02:55:13.096114 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096072 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 02:55:13.096114 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096084 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 02:55:13.096114 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096093 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 02:55:13.096114 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096109 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 02:55:13.096280 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096122 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 02:55:13.096615 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096605 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 02:55:13.096664 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.096620 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 02:55:13.100317 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.100302 2578 server.go:1263] "Started kubelet" Oct 30 02:55:13.100477 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.100419 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 02:55:13.100527 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.100504 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 02:55:13.101097 ip-10-0-142-236 systemd[1]: Started Kubernetes Kubelet. Oct 30 02:55:13.105216 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.105192 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 02:55:13.106014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.105935 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 02:55:13.109285 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.109022 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 02:55:13.109285 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.108600 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 02:55:13.112766 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.112744 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 02:55:13.113228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.113210 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 02:55:13.113339 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.113306 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.113852 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.113833 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 02:55:13.113852 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.113838 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 02:55:13.114009 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.113861 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 02:55:13.114009 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.113910 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 02:55:13.114009 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.113919 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 02:55:13.114315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114300 2578 factory.go:55] Registering systemd factory Oct 30 02:55:13.114315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114315 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 02:55:13.114530 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114515 2578 factory.go:153] Registering CRI-O factory Oct 30 02:55:13.114570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114535 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 02:55:13.114607 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114588 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 02:55:13.114664 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114616 2578 factory.go:103] Registering Raw factory Oct 30 02:55:13.114664 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.114631 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 02:55:13.115462 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.115448 2578 manager.go:319] Starting recovery of all containers Oct 30 02:55:13.126551 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.126536 2578 manager.go:324] Recovery completed Oct 30 02:55:13.131223 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.131183 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.132984 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.132945 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-236.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 02:55:13.133073 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.133031 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 02:55:13.133964 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.133933 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.134063 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.133985 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.134063 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.134001 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.134721 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.134702 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 02:55:13.134721 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.134718 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 02:55:13.134852 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.134737 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 02:55:13.135626 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.135607 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.136547 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.136525 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.136632 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.136561 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.136632 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.136582 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.137079 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.137065 2578 policy_none.go:50] "Start" Oct 30 02:55:13.137138 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.137104 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 02:55:13.137138 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.137117 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 02:55:13.137717 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.136294 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18e32f251ec786c2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-10-30 02:55:13.100277442 +0000 UTC m=+0.332666324,LastTimestamp:2026-10-30 02:55:13.100277442 +0000 UTC m=+0.332666324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}" Oct 30 02:55:13.139268 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.139236 2578 policy_none.go:44] "Start" Oct 30 02:55:13.162343 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.162244 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18e32f2520c9a6bd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-236.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-10-30 02:55:13.133971133 +0000 UTC m=+0.366360023,LastTimestamp:2026-10-30 02:55:13.133971133 +0000 UTC m=+0.366360023,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}" Oct 30 02:55:13.162715 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.162691 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-236.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 02:55:13.162787 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.162691 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 02:55:13.162787 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.162697 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 02:55:13.163449 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.163431 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 02:55:13.163525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.163493 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbvxs" Oct 30 02:55:13.170836 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.170808 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 02:55:13.173238 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173224 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 02:55:13.173331 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.173279 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 02:55:13.173331 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173292 2578 server.go:87] "Starting device plugin registration server" Oct 30 02:55:13.173533 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173520 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 02:55:13.173645 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173536 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 02:55:13.173645 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173628 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 02:55:13.173769 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173701 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 02:55:13.173769 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.173708 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 02:55:13.174282 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.174259 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 02:55:13.174365 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.174312 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.176171 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.176151 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sbvxs" Oct 30 02:55:13.177147 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.177072 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18e32f2520c9fa1a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-236.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-10-30 02:55:13.133992474 +0000 UTC m=+0.366381358,LastTimestamp:2026-10-30 02:55:13.133992474 +0000 UTC m=+0.366381358,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}" Oct 30 02:55:13.207605 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.194379 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-236.ec2.internal.18e32f2520ca342b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-236.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-236.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-236.ec2.internal,},FirstTimestamp:2026-10-30 02:55:13.134007339 +0000 UTC m=+0.366396222,LastTimestamp:2026-10-30 02:55:13.134007339 +0000 UTC m=+0.366396222,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-236.ec2.internal,}" Oct 30 02:55:13.233564 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.233404 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 02:55:13.234832 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.234808 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 02:55:13.234929 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.234837 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 02:55:13.234929 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.234859 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 02:55:13.234929 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.234896 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 02:55:13.251807 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.251789 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:55:13.274063 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.274037 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.275077 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.275058 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.275174 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.275088 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.275174 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.275105 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.275174 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.275131 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.296824 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.296773 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.296824 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.296795 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-236.ec2.internal\": node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.335436 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.335393 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"] Oct 30 02:55:13.335544 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.335473 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.336342 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.336324 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.336432 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.336357 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.336432 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.336368 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.338532 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.338520 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.338691 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.338675 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.338740 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.338710 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.338938 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.338926 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.339877 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.339861 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.339877 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.339868 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.339983 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.339886 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.339983 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.339892 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.339983 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.339898 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.339983 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.339902 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.341940 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.341924 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.342023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.341950 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:13.342729 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.342711 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:13.342809 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.342740 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:13.342809 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.342750 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:13.370568 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.370536 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.372448 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.372431 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-236.ec2.internal\" not found" node="ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.416024 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.416002 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.416122 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.416033 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.416122 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.416049 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c527eada420e7f4995c8eee6de2ab03-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"0c527eada420e7f4995c8eee6de2ab03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.439120 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.439099 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.516314 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.516289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.516405 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.516319 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.516405 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.516337 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c527eada420e7f4995c8eee6de2ab03-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"0c527eada420e7f4995c8eee6de2ab03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.516405 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.516374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c527eada420e7f4995c8eee6de2ab03-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"0c527eada420e7f4995c8eee6de2ab03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.516405 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.516378 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.516525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.516407 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.539393 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.539370 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.640297 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.640209 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.673559 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.673537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.675700 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:13.675682 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:55:13.740691 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.740656 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.841187 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.841165 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:13.941692 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:13.941620 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.251766 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.251741 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.261879 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.261855 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 02:55:15.261997 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.261981 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 02:55:15.352196 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.352165 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.390504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.390462 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:50:14 +0000 UTC" deadline="2028-04-05 02:23:58.789952827 +0000 UTC" Oct 30 02:55:15.390504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.390499 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12551h28m43.399457566s" Oct 30 02:55:15.422814 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.422788 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:55:15.423061 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.423039 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:55:15.424320 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.424302 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:55:15.424429 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.424333 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:55:15.424429 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.424347 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:55:15.433495 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:15.433472 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c527eada420e7f4995c8eee6de2ab03.slice/crio-5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd WatchSource:0}: Error finding container 5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd: Status 404 returned error can't find the container with id 5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd Oct 30 02:55:15.433766 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:15.433746 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c66d532a44fea0497f5e76e786c3710.slice/crio-7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68 WatchSource:0}: Error finding container 7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68: Status 404 returned error can't find the container with id 7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68 Oct 30 02:55:15.439765 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.439750 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 02:55:15.447741 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.447707 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"0c527eada420e7f4995c8eee6de2ab03","Type":"ContainerStarted","Data":"5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd"} Oct 30 02:55:15.448710 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.448688 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"8c66d532a44fea0497f5e76e786c3710","Type":"ContainerStarted","Data":"7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68"} Oct 30 02:55:15.452828 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.452800 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.553374 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.553338 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.632489 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.632460 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:55:15.653477 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.653443 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.753997 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.753925 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.854748 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.854719 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.955334 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:15.955307 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:55:15.962016 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:15.961984 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:55:16.024025 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.023943 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:55:16.048045 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.048013 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:55:16.049041 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.049011 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:55:16.070145 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.070116 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:55:16.298277 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.298195 2578 apiserver.go:52] "Watching apiserver" Oct 30 02:55:16.323802 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.323769 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 02:55:16.340076 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.340049 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 02:55:16.340860 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.340456 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-7p5r2","openshift-cluster-node-tuning-operator/tuned-nlh5q","openshift-dns/node-resolver-pkc7w","openshift-image-registry/node-ca-fsp8w","openshift-multus/multus-additional-cni-plugins-4t8gf","openshift-multus/network-metrics-daemon-zmq8r","openshift-network-diagnostics/network-check-target-knnkw","kube-system/konnectivity-agent-vhplh","openshift-multus/multus-zdmr4","openshift-ovn-kubernetes/ovnkube-node-lczmd","kube-system/global-pull-secret-syncer-569p6","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal"] Oct 30 02:55:16.343265 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.343091 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:16.343265 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.343168 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:16.345540 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.345520 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.347446 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.347425 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.349665 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.349645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.352004 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.351980 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.356148 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.356130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:16.356243 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.356203 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:16.358425 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.358401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.360467 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.360447 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.361133 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.361109 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.361232 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.361130 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.361938 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.361919 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lx7sd\"" Oct 30 02:55:16.362029 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.361969 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 02:55:16.363666 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.363645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.366564 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.364844 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.366564 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.364904 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 02:55:16.366564 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.364916 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.366564 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.365076 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.366564 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.365131 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qr586\"" Oct 30 02:55:16.367265 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367232 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 02:55:16.367356 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367261 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 02:55:16.367750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367557 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-726c9\"" Oct 30 02:55:16.367750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367644 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 02:55:16.367750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367657 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.367750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367665 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.367750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367721 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.367750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367743 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5kvz4\"" Oct 30 02:55:16.368030 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.367879 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.371005 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.370887 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.371005 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.370953 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:16.373721 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.373691 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 02:55:16.373897 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.373878 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.374004 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.373923 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-65dst\"" Oct 30 02:55:16.374696 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.374678 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 02:55:16.375749 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.375730 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-9cl95" Oct 30 02:55:16.375866 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.375763 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.388018 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.387943 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-9cl95" Oct 30 02:55:16.424846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.424808 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 02:55:16.438211 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438189 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-multus-certs\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.438341 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438222 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysconfig\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438341 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438247 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqhsc\" (UniqueName: \"kubernetes.io/projected/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-kube-api-access-wqhsc\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:16.438341 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438307 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-kubelet-config\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.438341 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438337 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5zqc\" (UniqueName: \"kubernetes.io/projected/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kube-api-access-p5zqc\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.438479 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438355 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.438479 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438396 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-sys\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438479 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438437 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-netns\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.438479 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438463 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.438622 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438494 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-multus\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.438622 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438531 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-kubernetes\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438622 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c32c037-b7a6-4a81-9ce9-478c53e023dd-agent-certs\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.438622 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438616 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.438793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438636 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-run\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438697 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-host\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-tuned\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d48d\" (UniqueName: \"kubernetes.io/projected/a577fb95-15eb-4954-91f1-426fca88fb5b-kube-api-access-7d48d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.438793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438771 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-bin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.439057 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438808 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf9c99a-3031-4ce6-b564-df6a1feda93c-host-slash\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.439057 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438875 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-ovn\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439057 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438917 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439057 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.438958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-bin\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439057 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439048 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439088 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvhbk\" (UniqueName: \"kubernetes.io/projected/5b610fff-d6f5-4856-a97c-04893bee2bb5-kube-api-access-jvhbk\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439144 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-k8s-cni-cncf-io\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439186 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-systemd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439208 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439226 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-node-log\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439242 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23217ed3-9c56-47de-be14-b34a31cc29e2-host\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.439262 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439258 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439276 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-registration-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-etc-kubernetes\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439367 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf9c99a-3031-4ce6-b564-df6a1feda93c-iptables-alerter-script\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439392 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovn-node-metrics-cert\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439433 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ef4e11-7e6a-49ad-a017-d51735f3b124-tmp-dir\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cnibin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.439518 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439475 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-conf-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-kubelet\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-env-overrides\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439588 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-device-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-cnibin\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439682 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpbxl\" (UniqueName: \"kubernetes.io/projected/9cf9c99a-3031-4ce6-b564-df6a1feda93c-kube-api-access-cpbxl\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439721 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-config\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439754 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mqw\" (UniqueName: \"kubernetes.io/projected/a9b5ab42-4a77-496d-bb0b-b1dd59680622-kube-api-access-g8mqw\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.439791 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-socket-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439834 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lll85\" (UniqueName: \"kubernetes.io/projected/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-kube-api-access-lll85\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439951 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-modprobe-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.439990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgnsr\" (UniqueName: \"kubernetes.io/projected/23217ed3-9c56-47de-be14-b34a31cc29e2-kube-api-access-cgnsr\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02ef4e11-7e6a-49ad-a017-d51735f3b124-hosts-file\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440109 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-system-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440132 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-log-socket\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440179 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-script-lib\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.440227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440213 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-dbus\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440248 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440277 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-sys-fs\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440302 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440342 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-systemd\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440361 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-var-lib-kubelet\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23217ed3-9c56-47de-be14-b34a31cc29e2-serviceca\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440450 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-hostroot\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-lib-modules\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440514 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-systemd-units\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440555 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jbh\" (UniqueName: \"kubernetes.io/projected/02ef4e11-7e6a-49ad-a017-d51735f3b124-kube-api-access-b9jbh\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.440596 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440594 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440611 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-socket-dir-parent\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440645 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-kubelet\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440680 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-netd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cni-binary-copy\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-netns\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440777 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-tmp\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-slash\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440950 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-var-lib-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.440976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c32c037-b7a6-4a81-9ce9-478c53e023dd-konnectivity-ca\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.441027 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-daemon-config\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.441071 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-conf\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.441129 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.441102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-etc-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.462717 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.462644 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 02:55:16.462717 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.462681 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 02:55:16.462717 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.462692 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.462986 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.462849 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 02:55:16.462986 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.462885 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.463089 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463019 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.463089 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463022 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 02:55:16.463089 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463048 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 02:55:16.463089 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463080 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 02:55:16.463257 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463092 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 02:55:16.463473 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463452 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pz2d6\"" Oct 30 02:55:16.463681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463653 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q5wfc\"" Oct 30 02:55:16.463760 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463667 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vcctk\"" Oct 30 02:55:16.463840 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463785 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q4rkf\"" Oct 30 02:55:16.463902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463836 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 02:55:16.463902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.463881 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 02:55:16.542231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542193 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-kubelet\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.542231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542237 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-netd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cni-binary-copy\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542285 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-netns\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542308 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-tmp\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542318 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-kubelet\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-netd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-slash\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542381 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-netns\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-var-lib-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542416 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-slash\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542426 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c32c037-b7a6-4a81-9ce9-478c53e023dd-konnectivity-ca\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-daemon-config\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.542463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542456 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-var-lib-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542572 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-conf\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542588 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-etc-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542602 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-multus-certs\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysconfig\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhsc\" (UniqueName: \"kubernetes.io/projected/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-kube-api-access-wqhsc\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542683 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-kubelet-config\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542680 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zqc\" (UniqueName: \"kubernetes.io/projected/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kube-api-access-p5zqc\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542748 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542775 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-sys\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542784 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-conf\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542832 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-netns\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.543023 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.542900 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysconfig\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.543632 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.543037 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cni-binary-copy\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.543632 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.543140 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c32c037-b7a6-4a81-9ce9-478c53e023dd-konnectivity-ca\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.543910 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.543215 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-netns\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.544378 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544340 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-daemon-config\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.544451 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544391 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544572 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-etc-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544633 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-kubelet-config\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544700 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-multus-certs\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544740 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544794 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-multus\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544809 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-sys\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544873 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-kubernetes\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544942 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-multus\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.544995 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-kubernetes\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545035 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c32c037-b7a6-4a81-9ce9-478c53e023dd-agent-certs\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545072 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545278 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545365 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-run\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545428 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-host\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545452 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-tuned\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.551974 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d48d\" (UniqueName: \"kubernetes.io/projected/a577fb95-15eb-4954-91f1-426fca88fb5b-kube-api-access-7d48d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545513 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-bin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545601 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf9c99a-3031-4ce6-b564-df6a1feda93c-host-slash\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-ovn\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545670 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545702 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-bin\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhbk\" (UniqueName: \"kubernetes.io/projected/5b610fff-d6f5-4856-a97c-04893bee2bb5-kube-api-access-jvhbk\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-k8s-cni-cncf-io\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-systemd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545931 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545959 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-node-log\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.545983 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23217ed3-9c56-47de-be14-b34a31cc29e2-host\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546012 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-registration-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546068 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-etc-kubernetes\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546093 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf9c99a-3031-4ce6-b564-df6a1feda93c-iptables-alerter-script\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.552851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546117 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovn-node-metrics-cert\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546143 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ef4e11-7e6a-49ad-a017-d51735f3b124-tmp-dir\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546170 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cnibin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546196 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-conf-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546222 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-kubelet\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-env-overrides\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546283 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546335 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-device-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-cnibin\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546537 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbxl\" (UniqueName: \"kubernetes.io/projected/9cf9c99a-3031-4ce6-b564-df6a1feda93c-kube-api-access-cpbxl\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-config\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546591 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mqw\" (UniqueName: \"kubernetes.io/projected/a9b5ab42-4a77-496d-bb0b-b1dd59680622-kube-api-access-g8mqw\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-socket-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lll85\" (UniqueName: \"kubernetes.io/projected/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-kube-api-access-lll85\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546703 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-modprobe-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546733 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:16.553654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546762 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnsr\" (UniqueName: \"kubernetes.io/projected/23217ed3-9c56-47de-be14-b34a31cc29e2-kube-api-access-cgnsr\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546791 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02ef4e11-7e6a-49ad-a017-d51735f3b124-hosts-file\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546834 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-system-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546858 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-log-socket\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-script-lib\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546915 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-dbus\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546941 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546964 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-sys-fs\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.546992 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-systemd\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547050 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-var-lib-kubelet\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547078 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23217ed3-9c56-47de-be14-b34a31cc29e2-serviceca\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547104 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547133 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-hostroot\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547163 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-lib-modules\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-systemd-units\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547220 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jbh\" (UniqueName: \"kubernetes.io/projected/02ef4e11-7e6a-49ad-a017-d51735f3b124-kube-api-access-b9jbh\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.554319 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547245 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547275 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-socket-dir-parent\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547452 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-socket-dir-parent\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547503 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-device-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547562 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-systemd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547588 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547604 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-cnibin\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547670 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-k8s-cni-cncf-io\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.547711 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.547801 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:17.047766376 +0000 UTC m=+3.070587403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547871 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-tmp\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547943 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.547991 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-node-log\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548025 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-socket-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548072 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf9c99a-3031-4ce6-b564-df6a1feda93c-host-slash\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548076 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-bin\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548081 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.554898 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548113 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-ovn\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548130 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-bin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548194 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-run\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548195 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-registration-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548290 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548324 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-etc-kubernetes\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548464 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-host\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548607 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-systemd-units\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548884 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-hostroot\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.549041 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-lib-modules\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.549157 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-config\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.549292 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ef4e11-7e6a-49ad-a017-d51735f3b124-tmp-dir\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.548031 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23217ed3-9c56-47de-be14-b34a31cc29e2-host\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.549667 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf9c99a-3031-4ce6-b564-df6a1feda93c-iptables-alerter-script\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.549745 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.549978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.555618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550104 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cnibin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-conf-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550204 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-kubelet\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550301 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-modprobe-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550356 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-sys-fs\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.550379 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-env-overrides\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550807 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.550894 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-systemd\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.551027 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:55:17.051004064 +0000 UTC m=+3.073825094 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551122 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02ef4e11-7e6a-49ad-a017-d51735f3b124-hosts-file\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551179 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-var-lib-kubelet\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-dbus\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551393 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-log-socket\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551383 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-system-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551656 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23217ed3-9c56-47de-be14-b34a31cc29e2-serviceca\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.556366 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.551710 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-script-lib\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.557084 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.554238 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-tuned\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.558843 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.558804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c32c037-b7a6-4a81-9ce9-478c53e023dd-agent-certs\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.559352 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.559326 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovn-node-metrics-cert\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.567227 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.567201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhsc\" (UniqueName: \"kubernetes.io/projected/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-kube-api-access-wqhsc\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:16.568371 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.568346 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zqc\" (UniqueName: \"kubernetes.io/projected/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kube-api-access-p5zqc\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.572633 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.572587 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d48d\" (UniqueName: \"kubernetes.io/projected/a577fb95-15eb-4954-91f1-426fca88fb5b-kube-api-access-7d48d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.572731 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.572694 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mqw\" (UniqueName: \"kubernetes.io/projected/a9b5ab42-4a77-496d-bb0b-b1dd59680622-kube-api-access-g8mqw\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.574432 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.574389 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lll85\" (UniqueName: \"kubernetes.io/projected/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-kube-api-access-lll85\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.574648 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.574626 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbxl\" (UniqueName: \"kubernetes.io/projected/9cf9c99a-3031-4ce6-b564-df6a1feda93c-kube-api-access-cpbxl\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.574762 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.574740 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhbk\" (UniqueName: \"kubernetes.io/projected/5b610fff-d6f5-4856-a97c-04893bee2bb5-kube-api-access-jvhbk\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.574832 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.574750 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnsr\" (UniqueName: \"kubernetes.io/projected/23217ed3-9c56-47de-be14-b34a31cc29e2-kube-api-access-cgnsr\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.577295 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.577262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jbh\" (UniqueName: \"kubernetes.io/projected/02ef4e11-7e6a-49ad-a017-d51735f3b124-kube-api-access-b9jbh\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.581142 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.581123 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:16.581220 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.581145 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:16.581220 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.581160 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:16.581287 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:16.581229 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:17.081214001 +0000 UTC m=+3.104035033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:16.638226 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.638194 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:55:16.665170 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.665128 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:55:16.670365 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.670341 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:55:16.677214 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.677188 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5034dc_e3d7_4af9_a808_d2ffad6ba46b.slice/crio-aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128 WatchSource:0}: Error finding container aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128: Status 404 returned error can't find the container with id aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128 Oct 30 02:55:16.678780 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.678759 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:55:16.681074 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.681022 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ef4e11_7e6a_49ad_a017_d51735f3b124.slice/crio-5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2 WatchSource:0}: Error finding container 5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2: Status 404 returned error can't find the container with id 5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2 Oct 30 02:55:16.689449 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.689426 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23217ed3_9c56_47de_be14_b34a31cc29e2.slice/crio-1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad WatchSource:0}: Error finding container 1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad: Status 404 returned error can't find the container with id 1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad Oct 30 02:55:16.701034 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.700977 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zdmr4" Oct 30 02:55:16.703544 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.703508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:55:16.705869 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.705850 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:55:16.710902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.710734 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:55:16.712268 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.712242 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc117c1fb_3ed4_4405_80a7_2fc1613bb98f.slice/crio-b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034 WatchSource:0}: Error finding container b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034: Status 404 returned error can't find the container with id b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034 Oct 30 02:55:16.715414 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.715397 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b610fff_d6f5_4856_a97c_04893bee2bb5.slice/crio-634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992 WatchSource:0}: Error finding container 634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992: Status 404 returned error can't find the container with id 634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992 Oct 30 02:55:16.720408 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.720381 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf9c99a_3031_4ce6_b564_df6a1feda93c.slice/crio-a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f WatchSource:0}: Error finding container a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f: Status 404 returned error can't find the container with id a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f Oct 30 02:55:16.725463 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.725435 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda577fb95_15eb_4954_91f1_426fca88fb5b.slice/crio-f5de401131219a9c69a722acefa0a0ee5e2dbb19b6707afcd36090fb89a558ab WatchSource:0}: Error finding container f5de401131219a9c69a722acefa0a0ee5e2dbb19b6707afcd36090fb89a558ab: Status 404 returned error can't find the container with id f5de401131219a9c69a722acefa0a0ee5e2dbb19b6707afcd36090fb89a558ab Oct 30 02:55:16.729170 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.729150 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:16.735360 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:16.735340 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:16.740191 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.740164 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c32c037_b7a6_4a81_9ce9_478c53e023dd.slice/crio-18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9 WatchSource:0}: Error finding container 18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9: Status 404 returned error can't find the container with id 18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9 Oct 30 02:55:16.745495 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:16.745469 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b5ab42_4a77_496d_bb0b_b1dd59680622.slice/crio-6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f WatchSource:0}: Error finding container 6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f: Status 404 returned error can't find the container with id 6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f Oct 30 02:55:17.050750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.050651 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:17.050905 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.050839 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:17.050966 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.050912 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:18.050892246 +0000 UTC m=+4.073713267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:17.152094 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.152064 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:17.152206 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.152116 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:17.152303 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.152260 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:17.152363 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.152320 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:55:18.152302847 +0000 UTC m=+4.175123854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:17.152746 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.152727 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:17.152850 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.152755 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:17.152850 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.152767 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:17.152966 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:17.152812 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:18.152798193 +0000 UTC m=+4.175619201 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:17.391715 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.391304 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:50:16 +0000 UTC" deadline="2028-06-02 14:10:05.347070897 +0000 UTC" Oct 30 02:55:17.391715 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.391586 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13955h14m47.955489449s" Oct 30 02:55:17.455435 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.455287 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" event={"ID":"a577fb95-15eb-4954-91f1-426fca88fb5b","Type":"ContainerStarted","Data":"f5de401131219a9c69a722acefa0a0ee5e2dbb19b6707afcd36090fb89a558ab"} Oct 30 02:55:17.462442 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.462370 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerStarted","Data":"634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992"} Oct 30 02:55:17.469482 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.469400 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zdmr4" event={"ID":"c117c1fb-3ed4-4405-80a7-2fc1613bb98f","Type":"ContainerStarted","Data":"b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034"} Oct 30 02:55:17.472618 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.472406 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fsp8w" event={"ID":"23217ed3-9c56-47de-be14-b34a31cc29e2","Type":"ContainerStarted","Data":"1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad"} Oct 30 02:55:17.474594 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.474567 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pkc7w" event={"ID":"02ef4e11-7e6a-49ad-a017-d51735f3b124","Type":"ContainerStarted","Data":"5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2"} Oct 30 02:55:17.476136 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.476111 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" event={"ID":"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b","Type":"ContainerStarted","Data":"aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128"} Oct 30 02:55:17.479161 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.479137 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"0c527eada420e7f4995c8eee6de2ab03","Type":"ContainerStarted","Data":"486bbd0e376b1d2bbe8ba6ea90a6c59bcf6d90a587f7860b872e08fb9afb144b"} Oct 30 02:55:17.481107 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.481070 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vhplh" event={"ID":"9c32c037-b7a6-4a81-9ce9-478c53e023dd","Type":"ContainerStarted","Data":"18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9"} Oct 30 02:55:17.482619 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.482577 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7p5r2" event={"ID":"9cf9c99a-3031-4ce6-b564-df6a1feda93c","Type":"ContainerStarted","Data":"a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f"} Oct 30 02:55:17.485487 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.485463 2578 generic.go:356] "Generic (PLEG): container finished" podID="8c66d532a44fea0497f5e76e786c3710" containerID="54cc8fae25303f61b5953e43282c355be3a44f100a436c3174b9e55da1553ae4" exitCode=0 Oct 30 02:55:17.485588 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.485563 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"8c66d532a44fea0497f5e76e786c3710","Type":"ContainerDied","Data":"54cc8fae25303f61b5953e43282c355be3a44f100a436c3174b9e55da1553ae4"} Oct 30 02:55:17.490136 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.490044 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f"} Oct 30 02:55:17.519679 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:17.518538 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" podStartSLOduration=1.518521986 podStartE2EDuration="1.518521986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:17.518509514 +0000 UTC m=+3.541330543" watchObservedRunningTime="2026-10-30 02:55:17.518521986 +0000 UTC m=+3.541343016" Oct 30 02:55:18.058221 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.058142 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:18.058394 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.058337 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:18.058453 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.058405 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:20.058382256 +0000 UTC m=+6.081203265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:18.158955 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.158922 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:18.159132 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.158973 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:18.159132 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.159119 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:18.159238 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.159177 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:55:20.159159717 +0000 UTC m=+6.181980745 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:18.159604 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.159581 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:18.159604 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.159606 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:18.159800 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.159620 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:18.159800 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.159662 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:20.159648327 +0000 UTC m=+6.182469348 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:18.391983 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.391898 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:50:16 +0000 UTC" deadline="2028-06-10 05:07:49.312907656 +0000 UTC" Oct 30 02:55:18.391983 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.391939 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14138h12m30.920974216s" Oct 30 02:55:18.449222 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.449193 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:18.449388 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.449326 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:18.449730 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.449713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:18.449856 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.449836 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:18.449926 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.449909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:18.449998 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:18.449980 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:18.517953 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:18.517871 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"8c66d532a44fea0497f5e76e786c3710","Type":"ContainerStarted","Data":"4cbc0fc332503e7a0b92995945059502f78e13a2b7056060cd98188857a234bd"} Oct 30 02:55:19.541189 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:19.541136 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="130e3f3e2c4515adc239ab0bb4c2bf4a34fb394e78182c646333fa500729ea4c" exitCode=0 Oct 30 02:55:19.542237 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:19.542209 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerDied","Data":"130e3f3e2c4515adc239ab0bb4c2bf4a34fb394e78182c646333fa500729ea4c"} Oct 30 02:55:19.615132 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:19.615032 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" podStartSLOduration=3.615014221 podStartE2EDuration="3.615014221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:18.560140005 +0000 UTC m=+4.582961051" watchObservedRunningTime="2026-10-30 02:55:19.615014221 +0000 UTC m=+5.637835231" Oct 30 02:55:20.074301 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.074223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:20.074458 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.074379 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:20.074458 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.074444 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:24.074425101 +0000 UTC m=+10.097246112 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:20.175149 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.175112 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:20.175322 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.175207 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:20.175383 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.175346 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:20.175383 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.175363 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:20.175383 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.175375 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:20.175551 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.175430 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:24.17541421 +0000 UTC m=+10.198235222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:20.175842 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.175809 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:20.175934 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.175881 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:55:24.175866093 +0000 UTC m=+10.198687104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:20.448121 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.448048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:20.448280 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.448186 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:20.448865 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.448648 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:20.448865 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.448755 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:20.449006 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.448910 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:20.449050 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:20.448999 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:20.556433 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:20.556393 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vhplh" event={"ID":"9c32c037-b7a6-4a81-9ce9-478c53e023dd","Type":"ContainerStarted","Data":"15ba27b5ae34aa458af7c47a995d5110d107f9d9b6bdf6121118b8a7f66c27fe"} Oct 30 02:55:21.564835 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:21.564749 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fsp8w" event={"ID":"23217ed3-9c56-47de-be14-b34a31cc29e2","Type":"ContainerStarted","Data":"43463717b2f58f9853e658cff41c59a56ca3b0de46c49f61bc12501c00bb0462"} Oct 30 02:55:21.605666 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:21.605608 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vhplh" podStartSLOduration=4.070755715 podStartE2EDuration="7.605586422s" totalImagesPullingTime="3.534830707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:20.598797633 +0000 UTC m=+6.621618663" watchObservedRunningTime="2026-10-30 02:55:21.605586422 +0000 UTC m=+7.628407456" Oct 30 02:55:22.447712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:22.446907 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:22.447712 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:22.447032 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:22.447712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:22.447409 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:22.447712 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:22.447513 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:22.447712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:22.447570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:22.447712 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:22.447641 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:23.193942 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:23.193308 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:23.195314 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:23.194486 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:23.240827 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:23.240755 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fsp8w" podStartSLOduration=4.711769966 podStartE2EDuration="9.240736097s" totalImagesPullingTime="4.528966131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:21.607004113 +0000 UTC m=+7.629825145" watchObservedRunningTime="2026-10-30 02:55:23.240736097 +0000 UTC m=+9.263557130" Oct 30 02:55:23.571808 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:23.571761 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" event={"ID":"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b","Type":"ContainerStarted","Data":"357bcfc8a79787684991c48da9e39b823960ce1c17a099451af174330fa7addd"} Oct 30 02:55:24.115571 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.115530 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:24.115733 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.115689 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:24.115803 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.115768 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:32.11574563 +0000 UTC m=+18.138566656 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.216701 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.216779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.216929 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.216981 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:55:32.216965819 +0000 UTC m=+18.239786843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.217072 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.217087 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.217099 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:24.217152 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.217130 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:32.217120433 +0000 UTC m=+18.239941454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:24.398215 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.398180 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 02:55:24.446312 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.444989 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:24.446312 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.446117 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:24.446312 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.446199 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:24.446312 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.446273 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:24.446771 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.446756 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:24.447003 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:24.446986 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:24.576102 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.576067 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="c2d61875c72c3565480d3b28ecf3021c0d1cda73eeee7ca592469c72f2127087" exitCode=0 Oct 30 02:55:24.576389 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.576367 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerDied","Data":"c2d61875c72c3565480d3b28ecf3021c0d1cda73eeee7ca592469c72f2127087"} Oct 30 02:55:24.579496 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.579430 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zdmr4" event={"ID":"c117c1fb-3ed4-4405-80a7-2fc1613bb98f","Type":"ContainerStarted","Data":"a1cc9ce98c1b742e3e79151b8b26080a55029cf4942d25ed0168b51f4b010883"} Oct 30 02:55:24.581481 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.581456 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-pkc7w" event={"ID":"02ef4e11-7e6a-49ad-a017-d51735f3b124","Type":"ContainerStarted","Data":"4de604f515f7b6c9168e232a66a5995161f3837bd1929fbae734d7af56b999aa"} Oct 30 02:55:24.584417 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.584386 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" event={"ID":"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b","Type":"ContainerStarted","Data":"2157b25e9fd3268d01f7ad2ebbc32cfe937eca0206904db518c93d4d538df25e"} Oct 30 02:55:24.692645 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.692547 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-pkc7w" podStartSLOduration=3.191607932 podStartE2EDuration="10.692535813s" totalImagesPullingTime="7.500927881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:24.692407853 +0000 UTC m=+10.715228883" watchObservedRunningTime="2026-10-30 02:55:24.692535813 +0000 UTC m=+10.715356843" Oct 30 02:55:24.735692 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:24.735643 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zdmr4" podStartSLOduration=3.269860517 podStartE2EDuration="10.735630114s" totalImagesPullingTime="7.465769597s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:24.735220219 +0000 UTC m=+10.758041250" watchObservedRunningTime="2026-10-30 02:55:24.735630114 +0000 UTC m=+10.758451143" Oct 30 02:55:25.389495 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.389387 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:55:24.398199684Z","UUID":"25c5cb95-5497-40a2-b4f9-6e7076c7e373","Handler":null,"Name":"","Endpoint":""} Oct 30 02:55:25.391188 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.391162 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 02:55:25.391336 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.391222 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 02:55:25.589370 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.588864 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" event={"ID":"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b","Type":"ContainerStarted","Data":"3af3793b8aec356c09668ab4d3e15ae95d1960ea2a8fa866885614cc7427fec6"} Oct 30 02:55:25.592921 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.592879 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="d8ee55609b99c76f98dabc05a8bce1f4cd58537ac9a96d6d2ea4da196dcfa0ef" exitCode=0 Oct 30 02:55:25.593771 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.593746 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerDied","Data":"d8ee55609b99c76f98dabc05a8bce1f4cd58537ac9a96d6d2ea4da196dcfa0ef"} Oct 30 02:55:25.711786 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:25.711687 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" podStartSLOduration=3.369586842 podStartE2EDuration="11.711667309s" totalImagesPullingTime="8.342080467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:25.649047033 +0000 UTC m=+11.671868064" watchObservedRunningTime="2026-10-30 02:55:25.711667309 +0000 UTC m=+11.734488339" Oct 30 02:55:26.444841 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.444758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:26.445345 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.444765 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:26.445345 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.444910 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:26.445345 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:26.444901 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:26.445345 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:26.445011 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:26.445345 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:26.445105 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:26.597785 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.597752 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="526501a468726f16618f5127787c77c54bc83cf150771d4b52d9c4fceb24b3af" exitCode=0 Oct 30 02:55:26.597959 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.597845 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerDied","Data":"526501a468726f16618f5127787c77c54bc83cf150771d4b52d9c4fceb24b3af"} Oct 30 02:55:26.600072 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.600051 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7p5r2" event={"ID":"9cf9c99a-3031-4ce6-b564-df6a1feda93c","Type":"ContainerStarted","Data":"16f2a8fd01687efdd372bdeae3143502f44dcffd0bae5e72e9f8a94c4688335e"} Oct 30 02:55:26.601625 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.601603 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" event={"ID":"a577fb95-15eb-4954-91f1-426fca88fb5b","Type":"ContainerStarted","Data":"95a02b862e0cf09f9e93728dce7bb2c90b09f3e5937e874df89561e90829710e"} Oct 30 02:55:26.751176 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.751082 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7p5r2" podStartSLOduration=4.889223364 podStartE2EDuration="12.7510646s" totalImagesPullingTime="7.861841236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:26.742150366 +0000 UTC m=+12.764971395" watchObservedRunningTime="2026-10-30 02:55:26.7510646 +0000 UTC m=+12.773885631" Oct 30 02:55:26.751336 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:26.751188 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" podStartSLOduration=2.979894043 podStartE2EDuration="12.751181s" totalImagesPullingTime="9.771286957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:26.700897372 +0000 UTC m=+12.723718402" watchObservedRunningTime="2026-10-30 02:55:26.751181 +0000 UTC m=+12.774002029" Oct 30 02:55:28.445323 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:28.445095 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:28.445890 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:28.445171 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:28.445890 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:28.445560 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:28.445890 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:28.445428 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:28.445890 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:28.445200 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:28.445890 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:28.445655 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:28.609298 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:28.608894 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="b3979947a7f62f1b28b2d9d9f4676037e7ae7ee31e0f8e652188c7f8fda646c7" exitCode=0 Oct 30 02:55:28.609298 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:28.608957 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerDied","Data":"b3979947a7f62f1b28b2d9d9f4676037e7ae7ee31e0f8e652188c7f8fda646c7"} Oct 30 02:55:29.615001 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:29.614969 2578 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="9bc97544f5a9f06b013ecc818b068daebb75f4f7b11779da15d8aacf44ee229e" exitCode=0 Oct 30 02:55:29.615478 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:29.615018 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerDied","Data":"9bc97544f5a9f06b013ecc818b068daebb75f4f7b11779da15d8aacf44ee229e"} Oct 30 02:55:30.119092 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.119058 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:30.119278 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.119217 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:55:30.119968 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.119944 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:55:30.445507 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.445388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:30.445507 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.445413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:30.445696 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.445527 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:30.445696 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:30.445527 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:30.445696 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:30.445654 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:30.445854 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:30.445737 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:30.620311 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.620270 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" event={"ID":"5b610fff-d6f5-4856-a97c-04893bee2bb5","Type":"ContainerStarted","Data":"ab21fc63f88eadbbbf67a5ad42cc3b02a21500ec444fccb848bf83265847f592"} Oct 30 02:55:30.693201 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:30.693154 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4t8gf" podStartSLOduration=6.848022348 podStartE2EDuration="16.693142074s" totalImagesPullingTime="9.845119726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:30.680742284 +0000 UTC m=+16.703563313" watchObservedRunningTime="2026-10-30 02:55:30.693142074 +0000 UTC m=+16.715963103" Oct 30 02:55:31.625121 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:31.624841 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"b3b2f3f62181e604092c9a4b2415416461f879a4f2c13d9b83f0099eadb68c69"} Oct 30 02:55:31.625630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:31.625153 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"78c3768ed4133a3921c20dc8fc435829df8790ad82797b87978c541933fa5f31"} Oct 30 02:55:31.625630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:31.625190 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"25511270b34762e07989e99be2faa35abfee9c1d5645f60a3233f626aa4b7a20"} Oct 30 02:55:31.625630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:31.625220 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"c6d4fcf57bd16dfdf2be7661e5393c5fdb655fc9a5d93b4f5d32b2575260ce08"} Oct 30 02:55:31.625630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:31.625246 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"d7536e61b09d4e71a5ccd6631318eef41dec2c0998f61c7c276d1ab1b8280a1b"} Oct 30 02:55:31.625630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:31.625272 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"4e26bb883a99aaad9f86c460048995f2dae65ddc16c90b0c4d68bf59c0d1d86e"} Oct 30 02:55:32.175653 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:32.175610 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:32.175796 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.175758 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:32.175860 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.175836 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:55:48.175807218 +0000 UTC m=+34.198628241 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:32.276894 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:32.276850 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:32.276894 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:32.276898 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:32.277091 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.276996 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:32.277091 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.277007 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:32.277091 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.277024 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:32.277091 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.277034 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:32.277091 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.277042 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:55:48.277030869 +0000 UTC m=+34.299851876 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:32.277091 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.277071 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:55:48.27705899 +0000 UTC m=+34.299879996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:32.445106 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:32.445022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:32.445106 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:32.445048 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:32.445314 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.445169 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:32.445314 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:32.445215 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:32.445314 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.445283 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:32.445469 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:32.445344 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:33.631867 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:33.631834 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"54e0df5d7393108616dee68834f28d2a3682b70ae56403fbd839204e373d3865"} Oct 30 02:55:34.446751 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:34.446718 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:34.446897 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:34.446802 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:34.446897 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:34.446849 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:34.446994 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:34.446907 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:34.446994 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:34.446920 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:34.447057 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:34.447037 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:35.640325 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:35.640284 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" event={"ID":"a9b5ab42-4a77-496d-bb0b-b1dd59680622","Type":"ContainerStarted","Data":"14402b5c76c05983b6da4732a13de21de3162e57b2191b6f92de86823a21287d"} Oct 30 02:55:35.640932 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:35.640558 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:35.657003 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:35.656979 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:35.734860 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:35.734791 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" podStartSLOduration=7.424191373 podStartE2EDuration="21.734775437s" totalImagesPullingTime="14.310584064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:55:35.734214912 +0000 UTC m=+21.757035955" watchObservedRunningTime="2026-10-30 02:55:35.734775437 +0000 UTC m=+21.757596490" Oct 30 02:55:36.445607 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:36.445570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:36.445607 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:36.445596 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:36.446005 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:36.445709 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:36.446005 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:36.445758 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:36.446005 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:36.445864 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:36.446005 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:36.445938 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:36.642971 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:36.642940 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:36.643352 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:36.643020 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:36.657935 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:36.657910 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:55:37.630146 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:37.630071 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-569p6"] Oct 30 02:55:37.630389 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:37.630178 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:37.630389 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:37.630270 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:37.633206 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:37.633180 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zmq8r"] Oct 30 02:55:37.633303 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:37.633274 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:37.633379 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:37.633363 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:37.633621 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:37.633602 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knnkw"] Oct 30 02:55:37.633682 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:37.633672 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:37.633750 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:37.633736 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:39.445255 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:39.445221 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:39.445255 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:39.445242 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:39.445784 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:39.445359 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:39.445784 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:39.445400 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:39.445784 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:39.445472 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:39.445784 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:39.445528 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:41.445139 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:41.445102 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:41.445513 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:41.445118 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:41.445513 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:41.445211 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:41.445513 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:41.445277 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:41.445513 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:41.445122 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:41.445513 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:41.445365 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:41.562686 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:41.562657 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pkc7w_02ef4e11-7e6a-49ad-a017-d51735f3b124/dns-node-resolver/0.log" Oct 30 02:55:42.497838 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:42.497749 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fsp8w_23217ed3-9c56-47de-be14-b34a31cc29e2/node-ca/0.log" Oct 30 02:55:43.445159 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:43.445124 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:43.445390 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:43.445167 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:43.445390 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:43.445250 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:43.445390 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:43.445259 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:43.445390 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:43.445317 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:43.445693 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:43.445417 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:45.444882 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:45.444696 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:45.445224 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:45.444709 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:45.445224 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:45.444962 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:45.445224 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:45.444717 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:45.445224 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:45.445031 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:45.445224 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:45.445090 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:47.445589 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:47.445560 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:47.446068 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:47.445653 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:47.446068 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:47.445659 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:47.446068 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:47.445731 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:47.446068 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:47.445741 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:47.446068 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:47.445846 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:48.179309 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:48.179276 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:48.179512 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.179418 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:48.179512 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.179480 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs podName:eabb308d-a5a1-4207-bc27-c81e6bb4b2ec nodeName:}" failed. No retries permitted until 2026-10-30 02:56:20.179465258 +0000 UTC m=+66.202286277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs") pod "network-metrics-daemon-zmq8r" (UID: "eabb308d-a5a1-4207-bc27-c81e6bb4b2ec") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 02:55:48.280400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:48.280369 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:48.280400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:48.280402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:48.280649 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.280492 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:48.280649 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.280541 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret podName:feaaf2fd-3a0d-46a8-9252-9ea1909150df nodeName:}" failed. No retries permitted until 2026-10-30 02:56:20.28052894 +0000 UTC m=+66.303349959 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret") pod "global-pull-secret-syncer-569p6" (UID: "feaaf2fd-3a0d-46a8-9252-9ea1909150df") : object "kube-system"/"original-pull-secret" not registered Oct 30 02:55:48.280649 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.280543 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 02:55:48.280649 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.280567 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 02:55:48.280649 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.280581 2578 projected.go:196] Error preparing data for projected volume kube-api-access-kkhbh for pod openshift-network-diagnostics/network-check-target-knnkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:48.280649 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:48.280648 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh podName:38027d42-0deb-420c-a777-546f37f24db9 nodeName:}" failed. No retries permitted until 2026-10-30 02:56:20.280631614 +0000 UTC m=+66.303452636 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-kkhbh" (UniqueName: "kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh") pod "network-check-target-knnkw" (UID: "38027d42-0deb-420c-a777-546f37f24db9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 02:55:49.444927 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:49.444894 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:49.444927 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:49.444915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:49.445351 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:49.445002 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:49.445351 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:49.445015 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:49.445351 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:49.445070 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:49.445351 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:49.445127 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:51.444837 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:51.444775 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:51.445252 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:51.444896 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:51.445252 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:51.445002 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:51.445252 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:51.444901 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:51.445252 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:51.444903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:51.445252 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:51.445113 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:53.445003 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:53.444962 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:53.445519 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:53.445054 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:53.445519 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:53.445072 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:53.445519 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:53.445167 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:53.445519 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:53.445235 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:53.445519 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:53.445283 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:55.445411 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:55.445380 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:55.445837 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:55.445386 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:55.445837 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:55.445476 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:55.445837 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:55.445545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:55.445837 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:55.445388 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:55.445837 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:55.445652 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:57.445144 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:57.445097 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:57.445144 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:57.445149 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:57.445569 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:57.445117 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:57.445569 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:57.445230 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zmq8r" podUID="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" Oct 30 02:55:57.445569 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:57.445280 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-569p6" podUID="feaaf2fd-3a0d-46a8-9252-9ea1909150df" Oct 30 02:55:57.445569 ip-10-0-142-236 kubenswrapper[2578]: E1030 02:55:57.445347 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-knnkw" podUID="38027d42-0deb-420c-a777-546f37f24db9" Oct 30 02:55:58.811861 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.811833 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeReady" Oct 30 02:55:58.812210 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.811961 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 02:55:58.878054 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.877979 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gp6sv"] Oct 30 02:55:58.879766 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.879750 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:58.885943 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.885917 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rflzz"] Oct 30 02:55:58.887654 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.887637 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:58.895911 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.895893 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 02:55:58.896161 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.896145 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cth5s\"" Oct 30 02:55:58.896274 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.896259 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 02:55:58.901329 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.901310 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hpxr7"] Oct 30 02:55:58.903921 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.903903 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:58.905430 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.905411 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gp6sv"] Oct 30 02:55:58.911606 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.911586 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rflzz"] Oct 30 02:55:58.925683 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.925655 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpxr7"] Oct 30 02:55:58.997199 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.997172 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 02:55:58.997328 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.997206 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 02:55:58.997328 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.997233 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 02:55:58.997431 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.997421 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 02:55:58.997467 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.997438 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 02:55:58.998031 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.998008 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 02:55:58.998220 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.998204 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ndvn4\"" Oct 30 02:55:58.998290 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.998253 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 02:55:58.998341 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:58.998334 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-44jtl\"" Oct 30 02:55:59.051718 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051688 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab8f2d0-573d-495b-9075-bced70a5f27b-tmp-dir\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.051870 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051728 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzhw\" (UniqueName: \"kubernetes.io/projected/0ab8f2d0-573d-495b-9075-bced70a5f27b-kube-api-access-4pzhw\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.051870 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051755 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.051953 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051890 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab8f2d0-573d-495b-9075-bced70a5f27b-config-volume\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.051953 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051914 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab8f2d0-573d-495b-9075-bced70a5f27b-metrics-tls\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.051953 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-data-volume\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.052060 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.051998 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjdmz\" (UniqueName: \"kubernetes.io/projected/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-api-access-vjdmz\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.052060 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.052032 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-crio-socket\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.052121 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.052078 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-cert\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.052152 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.052136 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2lxb\" (UniqueName: \"kubernetes.io/projected/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-kube-api-access-r2lxb\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.052183 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.052158 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.152708 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152614 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lxb\" (UniqueName: \"kubernetes.io/projected/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-kube-api-access-r2lxb\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.152708 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.152708 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152704 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab8f2d0-573d-495b-9075-bced70a5f27b-tmp-dir\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.153014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152725 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzhw\" (UniqueName: \"kubernetes.io/projected/0ab8f2d0-573d-495b-9075-bced70a5f27b-kube-api-access-4pzhw\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.153014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152746 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152767 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab8f2d0-573d-495b-9075-bced70a5f27b-config-volume\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.153014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152788 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab8f2d0-573d-495b-9075-bced70a5f27b-metrics-tls\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.153014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-data-volume\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.152991 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdmz\" (UniqueName: \"kubernetes.io/projected/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-api-access-vjdmz\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153317 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153037 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-crio-socket\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153317 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153074 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-cert\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.153317 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153151 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab8f2d0-573d-495b-9075-bced70a5f27b-tmp-dir\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.153317 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153183 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-data-volume\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153317 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153242 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-crio-socket\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153671 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153645 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.153671 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.153662 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab8f2d0-573d-495b-9075-bced70a5f27b-config-volume\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.156570 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.156540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab8f2d0-573d-495b-9075-bced70a5f27b-metrics-tls\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.156688 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.156615 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.156688 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.156666 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-cert\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.176563 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.176540 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzhw\" (UniqueName: \"kubernetes.io/projected/0ab8f2d0-573d-495b-9075-bced70a5f27b-kube-api-access-4pzhw\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.176914 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.176883 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdmz\" (UniqueName: \"kubernetes.io/projected/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-api-access-vjdmz\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.178351 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.178334 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lxb\" (UniqueName: \"kubernetes.io/projected/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-kube-api-access-r2lxb\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.191199 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.191177 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gp6sv" Oct 30 02:55:59.198482 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.198465 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:55:59.214239 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.214213 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:55:59.445672 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.445646 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:55:59.445797 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.445713 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:55:59.445868 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.445803 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:55:59.462020 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.461980 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pchn6\"" Oct 30 02:55:59.462020 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.462013 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s7n2w\"" Oct 30 02:55:59.462150 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.462033 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:55:59.462150 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.462032 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:55:59.462150 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.462085 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqw84\"" Oct 30 02:55:59.462253 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.462167 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:55:59.462253 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.462242 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:55:59.502787 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.502767 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rflzz"] Oct 30 02:55:59.513474 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.513451 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gp6sv"] Oct 30 02:55:59.514547 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:59.514525 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab8f2d0_573d_495b_9075_bced70a5f27b.slice/crio-e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe WatchSource:0}: Error finding container e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe: Status 404 returned error can't find the container with id e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe Oct 30 02:55:59.526904 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.526881 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hpxr7"] Oct 30 02:55:59.528092 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:55:59.528060 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc663f5b1_25fa_4b8a_8dc7_95f273fecd53.slice/crio-f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51 WatchSource:0}: Error finding container f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51: Status 404 returned error can't find the container with id f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51 Oct 30 02:55:59.696218 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.696134 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gp6sv" event={"ID":"0ab8f2d0-573d-495b-9075-bced70a5f27b","Type":"ContainerStarted","Data":"e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe"} Oct 30 02:55:59.697364 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.697340 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpxr7" event={"ID":"c663f5b1-25fa-4b8a-8dc7-95f273fecd53","Type":"ContainerStarted","Data":"f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51"} Oct 30 02:55:59.698695 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.698676 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rflzz" event={"ID":"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb","Type":"ContainerStarted","Data":"fe9a48f766dff5473fbaa92abdcc4e10306af3f73b5af6db76aace99ddafdb22"} Oct 30 02:55:59.698769 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:55:59.698712 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rflzz" event={"ID":"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb","Type":"ContainerStarted","Data":"79d4742442e84fb605bd48e0ff5b343cf44cbbb789374e473b6f346ac475d7cf"} Oct 30 02:56:00.705130 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:00.705089 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rflzz" event={"ID":"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb","Type":"ContainerStarted","Data":"fd9599eb534eb9a4e6753533792cb6484bb27f485e498f18706e5c852f0f7f87"} Oct 30 02:56:01.710089 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:01.710050 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gp6sv" event={"ID":"0ab8f2d0-573d-495b-9075-bced70a5f27b","Type":"ContainerStarted","Data":"e44b4ae1bc861571a90a0e7a3cb10c50b38fa9b1d01b49ca2a12a46f33ed7037"} Oct 30 02:56:01.710492 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:01.710154 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gp6sv" event={"ID":"0ab8f2d0-573d-495b-9075-bced70a5f27b","Type":"ContainerStarted","Data":"e561ea8e1cc95da5bb448c1be2d905255f534c4cc3e0127c652d8d915cd1ed22"} Oct 30 02:56:01.710492 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:01.710185 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:01.755278 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:01.755220 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-gp6sv" podStartSLOduration=2.039624349 podStartE2EDuration="3.755207742s" totalImagesPullingTime="1.715583393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:01.754389341 +0000 UTC m=+47.777210371" watchObservedRunningTime="2026-10-30 02:56:01.755207742 +0000 UTC m=+47.778028787" Oct 30 02:56:02.713658 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:02.713619 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hpxr7" event={"ID":"c663f5b1-25fa-4b8a-8dc7-95f273fecd53","Type":"ContainerStarted","Data":"43e64a567e91ddbb999656e1b8104e4c9b5e4775f089d953d876eca05701dd22"} Oct 30 02:56:02.715677 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:02.715649 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rflzz" event={"ID":"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb","Type":"ContainerStarted","Data":"9419a7ff7791ebd4c5229cd29c71a0230b4c276adb6c6145717d79f79ef49e78"} Oct 30 02:56:02.771867 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:02.771800 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hpxr7" podStartSLOduration=2.156004756 podStartE2EDuration="4.771789042s" totalImagesPullingTime="2.615784286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:02.771480274 +0000 UTC m=+48.794301302" watchObservedRunningTime="2026-10-30 02:56:02.771789042 +0000 UTC m=+48.794610068" Oct 30 02:56:02.822337 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:02.822284 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rflzz" podStartSLOduration=2.819752308 podStartE2EDuration="4.822271563s" totalImagesPullingTime="2.002519255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 02:55:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:02.821597973 +0000 UTC m=+48.844419003" watchObservedRunningTime="2026-10-30 02:56:02.822271563 +0000 UTC m=+48.845092592" Oct 30 02:56:05.039828 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.039780 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh"] Oct 30 02:56:05.041808 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.041791 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.075863 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.075837 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:56:05.075863 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.075854 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:56:05.076020 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.075937 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 02:56:05.076055 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.076026 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 02:56:05.076093 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.076083 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 02:56:05.076126 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.076114 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x28m9\"" Oct 30 02:56:05.088951 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.088928 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh"] Oct 30 02:56:05.095659 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.095640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.095741 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.095677 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4l8l\" (UniqueName: \"kubernetes.io/projected/4259b75f-532b-4e73-9d4e-0d3d72909feb-kube-api-access-p4l8l\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.095741 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.095707 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.095850 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.095801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259b75f-532b-4e73-9d4e-0d3d72909feb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.196087 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.196051 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.196216 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.196096 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l8l\" (UniqueName: \"kubernetes.io/projected/4259b75f-532b-4e73-9d4e-0d3d72909feb-kube-api-access-p4l8l\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.196216 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.196119 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.196216 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.196155 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259b75f-532b-4e73-9d4e-0d3d72909feb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.196916 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.196888 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259b75f-532b-4e73-9d4e-0d3d72909feb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.198571 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.198548 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.198571 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.198556 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.222550 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.222519 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l8l\" (UniqueName: \"kubernetes.io/projected/4259b75f-532b-4e73-9d4e-0d3d72909feb-kube-api-access-p4l8l\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.352727 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.352659 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:05.645913 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.645875 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh"] Oct 30 02:56:05.646058 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:05.646033 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4259b75f_532b_4e73_9d4e_0d3d72909feb.slice/crio-7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800 WatchSource:0}: Error finding container 7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800: Status 404 returned error can't find the container with id 7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800 Oct 30 02:56:05.723768 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:05.723736 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" event={"ID":"4259b75f-532b-4e73-9d4e-0d3d72909feb","Type":"ContainerStarted","Data":"7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800"} Oct 30 02:56:06.905852 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.905805 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7db894548b-j267d"] Oct 30 02:56:06.908122 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.908098 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:06.927020 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.926999 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 02:56:06.939679 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.939654 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gmr24\"" Oct 30 02:56:06.939679 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.939654 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 02:56:06.939862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.939724 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 02:56:06.939862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.939775 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 02:56:06.939862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.939777 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 02:56:06.939862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.939837 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 02:56:06.965634 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.965589 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 02:56:06.966553 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:06.966519 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db894548b-j267d"] Oct 30 02:56:07.009252 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009221 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009267 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009290 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009314 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009353 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009400 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.009653 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.009426 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.033301 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.033263 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 02:56:07.109675 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109641 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.109857 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109684 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.109857 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109718 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.109857 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.109857 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109754 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.109857 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109787 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.109857 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109808 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.110063 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.109860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.110116 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.110092 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.110601 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.110492 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.110601 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.110541 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.111076 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.111054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.111184 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.111054 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.112441 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.112412 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.112742 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.112718 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.137621 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.137598 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.220894 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.220774 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:07.523860 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.523836 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7db894548b-j267d"] Oct 30 02:56:07.525124 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:07.525094 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68dc07a5_b6fe_4cac_a23b_e753b406161c.slice/crio-0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2 WatchSource:0}: Error finding container 0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2: Status 404 returned error can't find the container with id 0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2 Oct 30 02:56:07.731193 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.731156 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" event={"ID":"4259b75f-532b-4e73-9d4e-0d3d72909feb","Type":"ContainerStarted","Data":"ee692bbe04dbea07c0ee44e0311ddb0e5a6e83573bb7fde620cde09f96a62362"} Oct 30 02:56:07.731363 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.731209 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" event={"ID":"4259b75f-532b-4e73-9d4e-0d3d72909feb","Type":"ContainerStarted","Data":"de939e7a68bbafec955e3e464475d9d07a9faafd7cd9b8a498b7d5480698483c"} Oct 30 02:56:07.732404 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.732378 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db894548b-j267d" event={"ID":"68dc07a5-b6fe-4cac-a23b-e753b406161c","Type":"ContainerStarted","Data":"0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2"} Oct 30 02:56:07.777522 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:07.777438 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" podStartSLOduration=1.303903573 podStartE2EDuration="2.777423327s" totalImagesPullingTime="1.473519754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:07.77678386 +0000 UTC m=+53.799604889" watchObservedRunningTime="2026-10-30 02:56:07.777423327 +0000 UTC m=+53.800244356" Oct 30 02:56:08.659734 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:08.659701 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:10.040458 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.040422 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ffn7c"] Oct 30 02:56:10.045348 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.045320 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.087968 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.087934 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc"] Oct 30 02:56:10.090190 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.090168 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.120630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.120601 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 02:56:10.120780 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.120692 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mdrmf\"" Oct 30 02:56:10.120780 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.120711 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 02:56:10.120997 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.120980 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5fp68\"" Oct 30 02:56:10.129457 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129421 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.129457 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129457 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129637 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129483 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-root\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129637 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129499 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-tls\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129637 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129546 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jv57\" (UniqueName: \"kubernetes.io/projected/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-kube-api-access-9jv57\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129637 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129600 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.129843 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129634 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-textfile\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129843 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129670 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-metrics-client-ca\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129843 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-sys\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.129843 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129779 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.130034 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129857 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.130034 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129876 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85d227a6-403a-4b90-9750-eb26433e0758-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.130034 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129908 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-wtmp\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.130034 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.129939 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.130034 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.130016 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8wsw\" (UniqueName: \"kubernetes.io/projected/85d227a6-403a-4b90-9750-eb26433e0758-kube-api-access-d8wsw\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.132600 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.132584 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 02:56:10.132682 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.132638 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 02:56:10.143831 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.143796 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc"] Oct 30 02:56:10.157508 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.157485 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 02:56:10.157626 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.157584 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 02:56:10.231109 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wsw\" (UniqueName: \"kubernetes.io/projected/85d227a6-403a-4b90-9750-eb26433e0758-kube-api-access-d8wsw\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.231231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231134 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.231231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231162 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-root\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231223 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-tls\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231247 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jv57\" (UniqueName: \"kubernetes.io/projected/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-kube-api-access-9jv57\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231269 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231291 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-textfile\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231302 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-root\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-metrics-client-ca\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231381 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-sys\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231415 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231412 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.231739 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231451 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231739 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-sys\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231739 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231471 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85d227a6-403a-4b90-9750-eb26433e0758-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.231739 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-wtmp\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.231739 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.232017 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.231799 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85d227a6-403a-4b90-9750-eb26433e0758-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.232315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.232127 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.232315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.232208 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.232315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.232291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-metrics-client-ca\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.232494 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.232418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-wtmp\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.232607 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.232584 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.232757 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.232686 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-textfile\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.234086 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.234059 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.234172 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.234139 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.234257 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.234236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-tls\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.234699 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.234680 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.257014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.256989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wsw\" (UniqueName: \"kubernetes.io/projected/85d227a6-403a-4b90-9750-eb26433e0758-kube-api-access-d8wsw\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.264373 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.264347 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jv57\" (UniqueName: \"kubernetes.io/projected/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-kube-api-access-9jv57\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.360323 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.360236 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:10.369793 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:10.369768 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25c69c2_ac9b_492c_9332_bd1e3abd1f1b.slice/crio-caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445 WatchSource:0}: Error finding container caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445: Status 404 returned error can't find the container with id caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445 Oct 30 02:56:10.405036 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.405007 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:10.725551 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.725525 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc"] Oct 30 02:56:10.726421 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:10.726390 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85d227a6_403a_4b90_9750_eb26433e0758.slice/crio-9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201 WatchSource:0}: Error finding container 9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201: Status 404 returned error can't find the container with id 9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201 Oct 30 02:56:10.743970 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.743940 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" event={"ID":"85d227a6-403a-4b90-9750-eb26433e0758","Type":"ContainerStarted","Data":"9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201"} Oct 30 02:56:10.745764 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.745729 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db894548b-j267d" event={"ID":"68dc07a5-b6fe-4cac-a23b-e753b406161c","Type":"ContainerStarted","Data":"58aa88fb7218ec27897780abbe8825f3835fe53c53917d85a9d24a8eef8e6a6d"} Oct 30 02:56:10.747009 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.746987 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffn7c" event={"ID":"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b","Type":"ContainerStarted","Data":"caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445"} Oct 30 02:56:10.796287 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:10.796242 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7db894548b-j267d" podStartSLOduration=2.16046498 podStartE2EDuration="4.796226082s" totalImagesPullingTime="2.635761102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:10.795187373 +0000 UTC m=+56.818008393" watchObservedRunningTime="2026-10-30 02:56:10.796226082 +0000 UTC m=+56.819047112" Oct 30 02:56:11.277874 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.277840 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:56:11.280464 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.280441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.310757 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.310375 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 02:56:11.310757 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.310487 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 02:56:11.310757 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.310513 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 02:56:11.310757 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.310654 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 02:56:11.310757 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.310718 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 02:56:11.311110 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.310766 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 02:56:11.311454 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.311430 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2cqx6\"" Oct 30 02:56:11.311642 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.311625 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 02:56:11.324284 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.324259 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 02:56:11.324421 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.324368 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 02:56:11.332982 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.332889 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:56:11.341235 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341375 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341375 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341375 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341313 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341375 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341375 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341346 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341375 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341369 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341389 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341416 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341434 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341451 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341469 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.341681 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.341487 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.442770 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442734 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.442950 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442781 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.442950 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442835 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.442950 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442866 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.442950 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442905 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.442990 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443015 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443041 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443071 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443107 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443402 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443148 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443402 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443174 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.443970 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.443619 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.444584 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.444553 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.445275 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.445248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.445665 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.445636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.445780 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.445761 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.446148 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.446129 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.446879 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.446804 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.447092 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.447035 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.447777 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.447752 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.447892 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.447760 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.448005 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.447985 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.448096 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.448073 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.485814 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.485779 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.628339 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.628297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:11.718179 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.718095 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:11.752808 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.752758 2578 generic.go:356] "Generic (PLEG): container finished" podID="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" containerID="09f200e56c45d031680836392306e859809e33fae0a98510a19ffd52c04e4103" exitCode=0 Oct 30 02:56:11.753009 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.752853 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffn7c" event={"ID":"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b","Type":"ContainerDied","Data":"09f200e56c45d031680836392306e859809e33fae0a98510a19ffd52c04e4103"} Oct 30 02:56:11.956277 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:11.956248 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd172e77b_1700_49d4_8513_07b3a516b821.slice/crio-6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb WatchSource:0}: Error finding container 6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb: Status 404 returned error can't find the container with id 6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb Oct 30 02:56:11.957957 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:11.957380 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:56:12.764835 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.764794 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffn7c" event={"ID":"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b","Type":"ContainerStarted","Data":"2e64d4322e4691f61c52c192fcf13560b8ac3f5f9a15251a22d8d14b95df5f89"} Oct 30 02:56:12.765291 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.764903 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ffn7c" event={"ID":"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b","Type":"ContainerStarted","Data":"d9d2fa8952348df78d231bfdd4c7a08f70028f8bc8d551301f22bb4ddb5abce2"} Oct 30 02:56:12.767469 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.767432 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" event={"ID":"85d227a6-403a-4b90-9750-eb26433e0758","Type":"ContainerStarted","Data":"77c19cf4268c928397dfcd01da9d9714c98b7cd93ef6747c1f6b9531342b7bc8"} Oct 30 02:56:12.767630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.767488 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" event={"ID":"85d227a6-403a-4b90-9750-eb26433e0758","Type":"ContainerStarted","Data":"8a6c1b16adb784ba8e6961f1b1b4e745dbf8387b42f6a0c3d027f38b7858fe19"} Oct 30 02:56:12.767630 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.767511 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" event={"ID":"85d227a6-403a-4b90-9750-eb26433e0758","Type":"ContainerStarted","Data":"99407d2881c80ee1bd79c93859cb3ddd6b5cc0674b8bd81d04ed9349befb092f"} Oct 30 02:56:12.768705 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.768679 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb"} Oct 30 02:56:12.812574 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.812533 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ffn7c" podStartSLOduration=1.991887889 podStartE2EDuration="2.812520369s" totalImagesPullingTime="739.928274ms" totalInitContainerRuntime="80.704206ms" isStatefulPod=true podCreationTimestamp="2026-10-30 02:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:12.811580106 +0000 UTC m=+58.834401134" watchObservedRunningTime="2026-10-30 02:56:12.812520369 +0000 UTC m=+58.835341398" Oct 30 02:56:12.880924 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:12.880881 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" podStartSLOduration=1.743393501 podStartE2EDuration="2.880867599s" totalImagesPullingTime="1.137474098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:12.880762467 +0000 UTC m=+58.903583498" watchObservedRunningTime="2026-10-30 02:56:12.880867599 +0000 UTC m=+58.903688629" Oct 30 02:56:13.773512 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:13.773478 2578 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02" exitCode=0 Oct 30 02:56:13.774065 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:13.773560 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02"} Oct 30 02:56:14.680784 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.680752 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-j69dd"] Oct 30 02:56:14.683049 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.683027 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.713346 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.713318 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-j69dd"] Oct 30 02:56:14.764922 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.764894 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rdfv4\"" Oct 30 02:56:14.765132 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.765116 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 02:56:14.775670 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.775646 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.776061 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.775786 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmtf\" (UniqueName: \"kubernetes.io/projected/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-kube-api-access-5nmtf\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.876789 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.876753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.877010 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.876980 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmtf\" (UniqueName: \"kubernetes.io/projected/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-kube-api-access-5nmtf\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.879832 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.879795 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.903542 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.903515 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmtf\" (UniqueName: \"kubernetes.io/projected/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-kube-api-access-5nmtf\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:14.994448 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:14.994413 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:15.316403 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.316380 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-j69dd"] Oct 30 02:56:15.316973 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:15.316948 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7fca2c_37ba_41b9_b68e_897f5fb0e8d9.slice/crio-16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8 WatchSource:0}: Error finding container 16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8: Status 404 returned error can't find the container with id 16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8 Oct 30 02:56:15.373067 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.373036 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f"] Oct 30 02:56:15.375664 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.375641 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.397123 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.397098 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 02:56:15.417686 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.417657 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 02:56:15.419737 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.419708 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f"] Oct 30 02:56:15.427251 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.427224 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 02:56:15.427365 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.427269 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 02:56:15.427365 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.427267 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 02:56:15.427365 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.427295 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 02:56:15.427365 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.427336 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s9wdq\"" Oct 30 02:56:15.482417 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482382 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-serving-certs-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482548 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482424 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2zzn\" (UniqueName: \"kubernetes.io/projected/62a522bc-31b4-477f-8a93-4d8959c0aabf-kube-api-access-t2zzn\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482548 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482453 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-metrics-client-ca\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482670 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482572 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482670 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482613 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482774 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482722 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-federate-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482774 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482757 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.482905 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.482801 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584382 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584310 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-serving-certs-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584382 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584360 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zzn\" (UniqueName: \"kubernetes.io/projected/62a522bc-31b4-477f-8a93-4d8959c0aabf-kube-api-access-t2zzn\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584581 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584465 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-metrics-client-ca\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584581 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584775 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584758 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-federate-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.584829 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584802 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.585766 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.584886 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.585766 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.585156 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-serving-certs-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.585766 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.585194 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-metrics-client-ca\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.586231 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.586201 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.587373 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.587330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-federate-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.587465 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.587414 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.587581 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.587536 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.587875 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.587454 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.608793 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.608767 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zzn\" (UniqueName: \"kubernetes.io/projected/62a522bc-31b4-477f-8a93-4d8959c0aabf-kube-api-access-t2zzn\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.689541 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.689508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:15.783616 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.783567 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" event={"ID":"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9","Type":"ContainerStarted","Data":"16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8"} Oct 30 02:56:15.790927 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.790899 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f"} Oct 30 02:56:15.791053 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.790965 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1"} Oct 30 02:56:15.791053 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.791002 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662"} Oct 30 02:56:15.791053 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.791033 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a"} Oct 30 02:56:15.791152 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:15.791072 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f"} Oct 30 02:56:16.033765 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.033689 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f"] Oct 30 02:56:16.034364 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:16.034331 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a522bc_31b4_477f_8a93_4d8959c0aabf.slice/crio-8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793 WatchSource:0}: Error finding container 8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793: Status 404 returned error can't find the container with id 8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793 Oct 30 02:56:16.102890 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.102851 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db894548b-j267d"] Oct 30 02:56:16.136027 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.136003 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58955f585-4thl7"] Oct 30 02:56:16.139463 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.139441 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.162299 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.162270 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58955f585-4thl7"] Oct 30 02:56:16.191315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191289 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191423 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191328 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191435 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191461 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191484 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191503 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191682 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191665 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.191749 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.191735 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292703 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292626 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292703 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292668 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292703 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292695 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292999 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292766 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292999 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292882 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292999 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.292999 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.292996 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.293186 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.293021 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.293902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.293306 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.293902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.293433 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.293902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.293538 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.293902 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.293858 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.294236 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.294210 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.295527 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.295507 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.295617 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.295526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.320398 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.320371 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.476828 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.476772 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:16.799326 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.799288 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerStarted","Data":"a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16"} Oct 30 02:56:16.800927 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.800853 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" event={"ID":"62a522bc-31b4-477f-8a93-4d8959c0aabf","Type":"ContainerStarted","Data":"8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793"} Oct 30 02:56:16.802447 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.802408 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" event={"ID":"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9","Type":"ContainerStarted","Data":"b6deb8b3587518c3f338cd9c60a5b034d8b7b162bfb9a6e9651431959f3823a8"} Oct 30 02:56:16.802604 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.802586 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:16.807313 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.807293 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58955f585-4thl7"] Oct 30 02:56:16.808567 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.808545 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:16.809061 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:16.809037 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a50d657_6d01_4305_b6f5_06eeaa081027.slice/crio-f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d WatchSource:0}: Error finding container f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d: Status 404 returned error can't find the container with id f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d Oct 30 02:56:16.883306 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.883241 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9255008890000003 podStartE2EDuration="5.883220988s" totalImagesPullingTime="2.945843508s" totalInitContainerRuntime="11.876591ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:16.87784781 +0000 UTC m=+62.900668840" watchObservedRunningTime="2026-10-30 02:56:16.883220988 +0000 UTC m=+62.906042019" Oct 30 02:56:16.978714 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:16.978653 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" podStartSLOduration=1.713154688 podStartE2EDuration="2.978627456s" totalImagesPullingTime="1.265472768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:16.959538148 +0000 UTC m=+62.982359205" watchObservedRunningTime="2026-10-30 02:56:16.978627456 +0000 UTC m=+63.001448483" Oct 30 02:56:17.221839 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.221788 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:17.440014 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.439978 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:56:17.443862 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.443836 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.494520 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.494452 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgf6s\"" Oct 30 02:56:17.495512 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.495424 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 02:56:17.496088 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.496063 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 02:56:17.496199 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.496172 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 02:56:17.496261 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.496218 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 02:56:17.501852 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.501812 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:56:17.506325 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506299 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506454 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506338 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506454 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506390 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506611 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506528 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506611 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506558 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506611 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506578 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506611 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506605 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506615 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d14jkodhae9m6\"" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506625 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506641 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506667 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506680 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506687 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506717 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506747 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506767 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506776 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506791 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506800 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506838 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 02:56:17.506851 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506850 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.507504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.506866 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.507504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.507000 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.507504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.507022 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.507504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.507039 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.507504 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.507125 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.523957 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.523935 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 02:56:17.523957 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.523950 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 02:56:17.524228 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.523967 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 02:56:17.524350 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.524227 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 02:56:17.537581 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.537555 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 02:56:17.537738 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.537596 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608371 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608418 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608450 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608468 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608484 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608527 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.608590 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608592 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608645 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608669 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608692 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608729 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608757 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608810 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608847 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608918 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608961 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.609044 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.608985 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.610003 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.609873 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.610127 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.610053 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.610444 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.610418 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.613612 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.612986 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.613612 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.613241 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.613612 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.613291 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.614737 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.613989 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.614737 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.614600 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.614737 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.614690 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.614955 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.614907 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.615031 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.615008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.615142 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.615014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.615409 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.615392 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.615750 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.615722 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.615888 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.615867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.615978 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.615945 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.616162 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.616144 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.616260 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.616236 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.617574 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.617555 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.647489 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.647461 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.776926 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.776891 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:17.808900 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.808868 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" event={"ID":"62a522bc-31b4-477f-8a93-4d8959c0aabf","Type":"ContainerStarted","Data":"b51e084788f5375ca62672969bdfe30f7bbf25928549ff238b574cec87890d6b"} Oct 30 02:56:17.814194 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.813188 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58955f585-4thl7" event={"ID":"2a50d657-6d01-4305-b6f5-06eeaa081027","Type":"ContainerStarted","Data":"68d8c78fd2c127de2aec31354da628787a22b15e7356616740c6596499eb6e2e"} Oct 30 02:56:17.814194 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.813234 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58955f585-4thl7" event={"ID":"2a50d657-6d01-4305-b6f5-06eeaa081027","Type":"ContainerStarted","Data":"f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d"} Oct 30 02:56:17.870695 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:17.870645 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58955f585-4thl7" podStartSLOduration=1.8706299400000002 podStartE2EDuration="1.87062994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:17.862207185 +0000 UTC m=+63.885028218" watchObservedRunningTime="2026-10-30 02:56:17.87062994 +0000 UTC m=+63.893451030" Oct 30 02:56:18.132970 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:18.132928 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3601216_ec2f_45de_9685_f97ac814e4a7.slice/crio-d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8 WatchSource:0}: Error finding container d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8: Status 404 returned error can't find the container with id d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8 Oct 30 02:56:18.133418 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.133395 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:56:18.816998 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.816960 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" event={"ID":"62a522bc-31b4-477f-8a93-4d8959c0aabf","Type":"ContainerStarted","Data":"44bd0e44f88d867cd5ea4f7035f3debc0b13f6271d010b6cfc786275cafb78f6"} Oct 30 02:56:18.817427 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.817027 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" event={"ID":"62a522bc-31b4-477f-8a93-4d8959c0aabf","Type":"ContainerStarted","Data":"1b69cdfb5f7ed9f78f62a0ea3ac77354266a76319600f8dc4911a7ca6cf380e5"} Oct 30 02:56:18.818419 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.818395 2578 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="c130d96630037f1c2d0f3333672ef1308f65196d728dd99ff514a37600d042ef" exitCode=0 Oct 30 02:56:18.818501 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.818482 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"c130d96630037f1c2d0f3333672ef1308f65196d728dd99ff514a37600d042ef"} Oct 30 02:56:18.818563 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.818551 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8"} Oct 30 02:56:18.872089 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:18.872037 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" podStartSLOduration=2.234024847 podStartE2EDuration="3.872021591s" totalImagesPullingTime="1.637996744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:18.871256545 +0000 UTC m=+64.894077588" watchObservedRunningTime="2026-10-30 02:56:18.872021591 +0000 UTC m=+64.894842620" Oct 30 02:56:20.239024 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.238982 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:20.253334 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.253303 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:56:20.261838 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.261787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:20.339919 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.339879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:20.340105 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.339935 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:20.354315 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.354287 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:56:20.354940 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.354892 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:56:20.363544 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.363522 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:20.375176 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.375139 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:56:20.385037 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.385008 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:20.471038 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.470996 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pchn6\"" Oct 30 02:56:20.477186 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.477159 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:20.478958 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.478938 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s7n2w\"" Oct 30 02:56:20.484127 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.484103 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqw84\"" Oct 30 02:56:20.485180 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.485161 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:20.491218 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.491196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:20.833689 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.833613 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-569p6"] Oct 30 02:56:20.846162 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:20.846131 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaaf2fd_3a0d_46a8_9252_9ea1909150df.slice/crio-ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31 WatchSource:0}: Error finding container ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31: Status 404 returned error can't find the container with id ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31 Oct 30 02:56:20.872394 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:20.872367 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38027d42_0deb_420c_a777_546f37f24db9.slice/crio-23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601 WatchSource:0}: Error finding container 23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601: Status 404 returned error can't find the container with id 23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601 Oct 30 02:56:20.872616 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.872596 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zmq8r"] Oct 30 02:56:20.873233 ip-10-0-142-236 kubenswrapper[2578]: W1030 02:56:20.873209 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabb308d_a5a1_4207_bc27_c81e6bb4b2ec.slice/crio-520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2 WatchSource:0}: Error finding container 520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2: Status 404 returned error can't find the container with id 520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2 Oct 30 02:56:20.874257 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:20.874199 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-knnkw"] Oct 30 02:56:21.320018 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.319678 2578 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58955f585-4thl7"] Oct 30 02:56:21.350846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.350114 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d9f9f777-vk5jc"] Oct 30 02:56:21.353580 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.353556 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.378759 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.378720 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9f9f777-vk5jc"] Oct 30 02:56:21.452543 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452510 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452569 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452621 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452640 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452712 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452921 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452718 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452921 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.452921 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.452789 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560333 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560434 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560472 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560514 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560571 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.560846 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.560612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.561810 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.561776 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.562259 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.562234 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.562752 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.562725 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.562927 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.562905 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.563390 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.563358 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.565029 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.564975 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.581525 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.581466 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.595968 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.595941 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.668809 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.668771 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:21.833732 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.833650 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-569p6" event={"ID":"feaaf2fd-3a0d-46a8-9252-9ea1909150df","Type":"ContainerStarted","Data":"ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31"} Oct 30 02:56:21.837134 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.837106 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmq8r" event={"ID":"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec","Type":"ContainerStarted","Data":"520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2"} Oct 30 02:56:21.839235 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:21.839205 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knnkw" event={"ID":"38027d42-0deb-420c-a777-546f37f24db9","Type":"ContainerStarted","Data":"23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601"} Oct 30 02:56:22.013965 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.001004 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d9f9f777-vk5jc"] Oct 30 02:56:22.849383 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.849349 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"1aa8707385d532c3f8503d01f2a5a70ab1bfad55f712a2d3373f55473ac98811"} Oct 30 02:56:22.849758 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.849460 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"5272a50ba406c489ad991ae1b2ebf6fb377de06426bfc6f3ea5f85c9d7ba2b50"} Oct 30 02:56:22.853723 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.853691 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmq8r" event={"ID":"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec","Type":"ContainerStarted","Data":"c6f1ecda89756f2552cfca8f59148ab56d5c05626d8399407ff10929ae270c90"} Oct 30 02:56:22.853839 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.853752 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zmq8r" event={"ID":"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec","Type":"ContainerStarted","Data":"79fbbfcf7df8f10e4fdc94d1201ccc5f69889345fef36e6b8e989093e5c14e57"} Oct 30 02:56:22.856434 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.856412 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9f777-vk5jc" event={"ID":"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14","Type":"ContainerStarted","Data":"79b81e373dcda5fc834fea15a76e34d376ab42aabb61a406f7c012b10a8f981a"} Oct 30 02:56:22.856532 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.856448 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9f777-vk5jc" event={"ID":"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14","Type":"ContainerStarted","Data":"e345b3a0960da3a4969a9f4c307169a770db0fe50b76f505c5b8dd3210ef79d1"} Oct 30 02:56:22.900286 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:22.900176 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zmq8r" podStartSLOduration=67.552716518 podStartE2EDuration="1m8.900158926s" totalImagesPullingTime="1.347442408s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:22.899285247 +0000 UTC m=+68.922106283" watchObservedRunningTime="2026-10-30 02:56:22.900158926 +0000 UTC m=+68.922979955" Oct 30 02:56:23.032670 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:23.032612 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5d9f9f777-vk5jc" podStartSLOduration=2.03259826 podStartE2EDuration="2.03259826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:23.029319604 +0000 UTC m=+69.052140634" watchObservedRunningTime="2026-10-30 02:56:23.03259826 +0000 UTC m=+69.055419289" Oct 30 02:56:25.875505 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.875454 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"7769afc9b2ee8d7341bd1b4999df938d305e716bc5bd4b7459339227a6160c90"} Oct 30 02:56:25.875996 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.875592 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"6f276f039487ef0d651cf4b887834990593214ef53c7630e57d1e3d9272bfddf"} Oct 30 02:56:25.875996 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.875629 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"86daec9c600d1147b4b63e67b1c5fe2fc502a28c556484ad2ba42dbc9bd9bd58"} Oct 30 02:56:25.875996 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.875662 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerStarted","Data":"f7bf51768c919abc553d06047686fdec9d20117b5ada1c87205d112103c887ed"} Oct 30 02:56:25.877327 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.877301 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-knnkw" event={"ID":"38027d42-0deb-420c-a777-546f37f24db9","Type":"ContainerStarted","Data":"7d66a0f552842a9ee56d404f3ff8b30689b324b177f42a1bc751aba27d406dad"} Oct 30 02:56:25.877436 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.877394 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:25.987022 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:25.986960 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.829916053 podStartE2EDuration="8.986938406s" totalImagesPullingTime="6.138881764s" totalInitContainerRuntime="18.140589ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:56:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:56:25.97357879 +0000 UTC m=+71.996399839" watchObservedRunningTime="2026-10-30 02:56:25.986938406 +0000 UTC m=+72.009759693" Oct 30 02:56:26.477150 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:26.477112 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:26.832306 ip-10-0-142-236 kubenswrapper[2578]: I1030 02:56:26.832186 2578 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:56:26.832357 ip-10-0-142-236 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 02:56:26.852210 ip-10-0-142-236 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 02:56:26.852376 ip-10-0-142-236 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 02:56:26.852423 ip-10-0-142-236 systemd[1]: kubelet.service: Consumed 4.401s CPU time. Oct 30 02:56:26.869359 ip-10-0-142-236 systemd[1]: Starting Kubernetes Kubelet... Oct 30 02:56:26.930962 ip-10-0-142-236 kubenswrapper[5902]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:56:26.930962 ip-10-0-142-236 kubenswrapper[5902]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 02:56:26.930962 ip-10-0-142-236 kubenswrapper[5902]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:56:26.930962 ip-10-0-142-236 kubenswrapper[5902]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 02:56:26.933042 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.932960 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.933042 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933041 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933046 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933050 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933053 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933055 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933058 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933061 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933068 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933071 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933075 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933079 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933082 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933085 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933088 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933091 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933094 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933097 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933099 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933102 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.933107 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933104 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933107 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933109 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933112 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933115 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933118 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933121 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933124 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933126 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933129 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933131 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933134 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933136 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933139 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933143 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933145 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933148 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933151 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933153 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933157 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.933563 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933159 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933162 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933164 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933167 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933169 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933172 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933174 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933177 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933179 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933182 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933185 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933189 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933191 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933194 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933197 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933199 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933202 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933205 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933207 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933210 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.934053 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933213 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933215 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933218 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933221 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933224 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933226 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933229 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933231 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933234 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933237 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933239 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933242 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933245 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933247 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933250 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933253 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933256 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933260 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933264 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.934565 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933267 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933270 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933273 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933276 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933278 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933281 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933284 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933286 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933289 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933291 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933294 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933297 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933300 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933302 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933305 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933307 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933310 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933312 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933315 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.935055 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933317 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933320 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933323 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933326 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933329 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933332 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933334 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.933337 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934108 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934115 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934118 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934121 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934124 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934127 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934130 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934133 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934135 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934138 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934141 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934144 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.935542 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934147 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934150 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934153 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934155 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934158 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934161 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934164 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934166 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934169 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934171 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934174 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934176 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934179 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934181 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934184 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934186 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934189 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934192 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934195 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934198 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.936038 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934201 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934204 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934207 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934209 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934212 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934214 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934217 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934219 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934222 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934224 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934227 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934230 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934233 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934235 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934239 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934247 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934250 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934253 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934255 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.936510 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934258 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934261 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934263 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934266 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934268 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934271 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934274 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934276 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934279 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934281 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934284 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934286 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934289 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934291 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934294 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934297 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934300 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934304 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934308 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.936964 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934311 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934314 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934316 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934319 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934321 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934323 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934326 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934329 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934332 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934336 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934338 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934341 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934344 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934346 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934349 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934351 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934354 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934357 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934360 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.937427 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934363 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934365 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934368 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934370 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934373 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934375 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934378 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934381 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934383 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934386 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934389 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934392 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934396 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934398 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934401 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934404 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934406 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934949 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934956 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934959 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.937889 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934962 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934965 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934968 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934971 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934973 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934976 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934979 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934982 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934991 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934994 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934996 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.934999 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935002 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935005 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935007 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935010 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935012 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935015 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935017 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935020 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.938372 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935022 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935026 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935029 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935031 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935034 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935037 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935039 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935042 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935045 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935048 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935050 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935053 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935056 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935059 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935062 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935065 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935068 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935070 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935073 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935075 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.938879 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935078 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935080 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935083 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935087 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935091 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935094 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935097 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935100 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935102 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935105 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935108 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935110 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935113 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935115 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935119 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935121 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935127 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935131 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935134 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.939493 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935137 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935140 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935143 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935145 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935149 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935153 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935156 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935159 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935161 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935164 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935167 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935169 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935172 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935175 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935177 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935180 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935182 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935185 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935187 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935190 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.940295 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935192 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935195 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935198 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935200 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935203 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935205 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935208 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935210 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935213 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935217 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935220 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935222 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935225 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935227 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935230 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935232 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935235 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935239 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935241 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935244 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.940833 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935247 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935249 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935252 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.935254 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935346 5902 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935353 5902 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935359 5902 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935364 5902 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935369 5902 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935372 5902 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935376 5902 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935381 5902 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935384 5902 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935387 5902 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935391 5902 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935394 5902 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935397 5902 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935400 5902 flags.go:64] FLAG: --cgroup-root="" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935403 5902 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935406 5902 flags.go:64] FLAG: --client-ca-file="" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935409 5902 flags.go:64] FLAG: --cloud-provider="external" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935414 5902 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935418 5902 flags.go:64] FLAG: --cluster-domain="" Oct 30 02:56:26.941308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935421 5902 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935424 5902 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935428 5902 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935432 5902 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935436 5902 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935438 5902 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935441 5902 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935445 5902 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935448 5902 flags.go:64] FLAG: --contention-profiling="false" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935451 5902 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935454 5902 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935457 5902 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935460 5902 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935465 5902 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935468 5902 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935471 5902 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935474 5902 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935477 5902 flags.go:64] FLAG: --enable-server="true" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935480 5902 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935484 5902 flags.go:64] FLAG: --event-burst="100" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935487 5902 flags.go:64] FLAG: --event-qps="50" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935490 5902 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935493 5902 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935496 5902 flags.go:64] FLAG: --eviction-hard="" Oct 30 02:56:26.941863 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935500 5902 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935503 5902 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935507 5902 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935510 5902 flags.go:64] FLAG: --eviction-soft="" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935512 5902 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935516 5902 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935520 5902 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935523 5902 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935526 5902 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935529 5902 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935534 5902 flags.go:64] FLAG: --feature-gates="" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935538 5902 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935541 5902 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935544 5902 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935548 5902 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935551 5902 flags.go:64] FLAG: --healthz-port="10248" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935555 5902 flags.go:64] FLAG: --help="false" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935558 5902 flags.go:64] FLAG: --hostname-override="ip-10-0-142-236.ec2.internal" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935561 5902 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935564 5902 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935567 5902 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935571 5902 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935574 5902 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935577 5902 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 02:56:26.942440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935580 5902 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935583 5902 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935598 5902 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935603 5902 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935606 5902 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935609 5902 flags.go:64] FLAG: --kube-reserved="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935612 5902 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935615 5902 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935618 5902 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935621 5902 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935623 5902 flags.go:64] FLAG: --lock-file="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935626 5902 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935629 5902 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935632 5902 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935640 5902 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935643 5902 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935647 5902 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935650 5902 flags.go:64] FLAG: --logging-format="text" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935657 5902 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935660 5902 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935663 5902 flags.go:64] FLAG: --manifest-url="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935666 5902 flags.go:64] FLAG: --manifest-url-header="" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935670 5902 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935674 5902 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935678 5902 flags.go:64] FLAG: --max-pods="110" Oct 30 02:56:26.943065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935681 5902 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935684 5902 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935687 5902 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935690 5902 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935693 5902 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935696 5902 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935699 5902 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935706 5902 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935709 5902 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935712 5902 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935715 5902 flags.go:64] FLAG: --pod-cidr="" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935718 5902 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935721 5902 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935724 5902 flags.go:64] FLAG: --pods-per-core="0" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935727 5902 flags.go:64] FLAG: --port="10250" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935730 5902 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935733 5902 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09939771943171ea3" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935736 5902 flags.go:64] FLAG: --qos-reserved="" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935739 5902 flags.go:64] FLAG: --read-only-port="10255" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935743 5902 flags.go:64] FLAG: --register-node="true" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935746 5902 flags.go:64] FLAG: --register-with-taints="" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935751 5902 flags.go:64] FLAG: --registry-burst="10" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935754 5902 flags.go:64] FLAG: --registry-qps="5" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935757 5902 flags.go:64] FLAG: --reserved-cpus="" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935761 5902 flags.go:64] FLAG: --reserved-memory="" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935766 5902 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 02:56:26.943646 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935769 5902 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935772 5902 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935775 5902 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935778 5902 flags.go:64] FLAG: --runonce="false" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935781 5902 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935784 5902 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935787 5902 flags.go:64] FLAG: --seccomp-default="false" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935790 5902 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935793 5902 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935796 5902 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935799 5902 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935802 5902 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935805 5902 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935808 5902 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935811 5902 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935826 5902 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935829 5902 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935832 5902 flags.go:64] FLAG: --system-cgroups="" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935835 5902 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935841 5902 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935844 5902 flags.go:64] FLAG: --tls-cert-file="" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935847 5902 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935851 5902 flags.go:64] FLAG: --tls-min-version="" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935854 5902 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935857 5902 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 02:56:26.944273 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935860 5902 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935863 5902 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935867 5902 flags.go:64] FLAG: --v="2" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935872 5902 flags.go:64] FLAG: --version="false" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935876 5902 flags.go:64] FLAG: --vmodule="" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935880 5902 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.935885 5902 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936025 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936029 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936032 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936035 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936038 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936041 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936043 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936046 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936049 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936052 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936055 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936057 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936060 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936063 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.945020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936065 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936068 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936071 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936074 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936076 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936079 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936081 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936084 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936086 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936090 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936092 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936095 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936097 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936101 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936104 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936107 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936109 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936113 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936117 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.945516 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936120 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936122 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936125 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936127 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936130 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936133 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936135 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936138 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936141 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936143 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936146 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936148 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936151 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936153 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936156 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936158 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936161 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936164 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936166 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936169 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.945986 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936173 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936176 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936179 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936182 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936184 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936187 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936191 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936194 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936197 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936200 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936204 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936206 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936212 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936215 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936218 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936221 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936223 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936226 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936229 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936231 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.946456 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936234 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936236 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936239 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936242 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936244 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936247 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936250 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936252 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936255 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936258 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936260 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936263 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936266 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936268 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936271 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936273 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936276 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936278 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936282 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.947009 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936285 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936287 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936290 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936293 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936296 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936298 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936301 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936304 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936307 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936309 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936312 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936314 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936317 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936321 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.936347 5902 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936384 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.947474 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936388 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936391 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936394 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936397 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936399 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936402 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936405 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936407 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936410 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936413 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936415 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936418 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936421 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936423 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936426 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936429 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936433 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936435 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936438 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.947936 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936442 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936445 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936447 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936450 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936453 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936456 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936458 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936461 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936463 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936467 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936472 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936475 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936478 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936480 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936483 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936485 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936488 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936491 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936493 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936496 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.948415 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936498 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936501 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936504 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936508 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936510 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936513 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936516 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936519 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936522 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936525 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936527 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936530 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936535 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936537 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936540 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936543 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936545 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936548 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936551 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.948948 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936553 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936556 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936558 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936561 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936564 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936566 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936569 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936571 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936574 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936576 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936579 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936581 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936584 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936586 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936589 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936591 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936594 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936596 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936599 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936601 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.949524 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936604 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936607 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936611 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936615 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936618 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936622 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936625 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936627 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936630 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936633 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936635 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936638 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936641 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936643 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936646 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936648 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936651 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936653 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936656 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.950070 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936659 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936661 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936664 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936666 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936669 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936671 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936674 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.936678 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.941810 5902 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.941846 5902 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.941870 5902 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.941875 5902 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941936 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941942 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941945 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941948 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941952 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941954 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941957 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.950581 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941960 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941963 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941966 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941968 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941971 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941973 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941976 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941979 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941982 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941984 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941987 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941990 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941993 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941996 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.941998 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942001 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942003 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942006 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942008 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.951332 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942011 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942015 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942017 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942020 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942023 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942027 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942032 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942035 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942037 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942040 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942043 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942045 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942048 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942050 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942053 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942056 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942058 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942062 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942064 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942067 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.951887 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942071 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942073 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942077 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942079 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942082 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942085 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942088 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942091 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942093 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942096 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942099 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942102 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942105 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942108 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942110 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942113 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942115 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942118 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942121 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942123 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.952373 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942126 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942128 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942131 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942134 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942137 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942139 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942142 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942145 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942148 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942150 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942154 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942158 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942161 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942170 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942173 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942175 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942178 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942181 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942183 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.952859 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942186 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942188 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942191 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942193 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942196 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942200 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942202 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942205 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942208 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942210 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942213 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942215 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942218 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942220 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942223 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942225 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942228 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942231 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942233 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.953311 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942235 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942238 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.942268 5902 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942405 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942409 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942412 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942415 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942418 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942427 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942430 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942433 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942436 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942439 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942441 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942444 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942447 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.953757 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942450 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942453 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942456 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942459 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942461 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942464 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942466 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942469 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942473 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942476 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942479 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942481 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942484 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942486 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942489 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942492 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942494 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942497 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942500 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.954153 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942503 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942505 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942508 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942510 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942513 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942515 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942519 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942522 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942524 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942528 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942532 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942535 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942537 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942540 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942543 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942545 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942548 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942550 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942553 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.954850 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942555 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942558 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942560 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942563 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942565 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942568 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942571 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942573 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942576 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942578 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942581 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942583 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942586 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942589 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942592 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942594 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942597 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942599 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942602 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942604 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.955619 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942607 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942610 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942613 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942615 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942618 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942620 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942623 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942626 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942628 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942631 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942633 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942636 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942638 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942641 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942643 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942646 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942649 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942651 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942654 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942656 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.956404 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942659 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942662 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942664 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942667 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942669 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942672 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942674 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942677 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942680 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942682 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942685 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942687 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942690 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942693 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942695 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.942718 5902 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 02:56:26.956921 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942749 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942755 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942759 5902 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942761 5902 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942765 5902 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942768 5902 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942771 5902 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942773 5902 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942776 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942778 5902 feature_gate.go:386] unrecognized feature gate: Example Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942781 5902 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942784 5902 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942786 5902 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942789 5902 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942792 5902 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942794 5902 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942797 5902 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942800 5902 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942802 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 02:56:26.957350 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942805 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942807 5902 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942810 5902 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942813 5902 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942838 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942841 5902 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942843 5902 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942846 5902 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942849 5902 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942851 5902 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942855 5902 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942858 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942861 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942864 5902 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942869 5902 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942871 5902 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942874 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942877 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942879 5902 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942882 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 02:56:26.958020 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942884 5902 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942887 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942890 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942893 5902 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942896 5902 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942898 5902 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942901 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942903 5902 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942906 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942909 5902 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942911 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942914 5902 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942933 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942937 5902 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942940 5902 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942942 5902 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942945 5902 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942948 5902 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942951 5902 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 02:56:26.958653 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942953 5902 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942958 5902 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942961 5902 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942964 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942967 5902 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942971 5902 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942974 5902 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942977 5902 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942980 5902 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942983 5902 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942986 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942988 5902 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942991 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942994 5902 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942996 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.942999 5902 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943001 5902 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943004 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943007 5902 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 02:56:26.959304 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943009 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943012 5902 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943015 5902 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943019 5902 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943022 5902 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943025 5902 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943028 5902 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943030 5902 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943033 5902 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943035 5902 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943038 5902 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943040 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943043 5902 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943045 5902 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943049 5902 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943052 5902 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943055 5902 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943057 5902 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943060 5902 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 02:56:26.959829 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943064 5902 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943068 5902 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943071 5902 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943073 5902 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943076 5902 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943080 5902 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943083 5902 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943085 5902 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943088 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:56:26.943090 5902 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.943145 5902 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944120 5902 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944176 5902 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944361 5902 server.go:1014] "Starting client certificate rotation" Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944513 5902 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944589 5902 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 02:50:14 +0000 UTC" deadline="2028-06-28 03:52:35.233941086 +0000 UTC" Oct 30 02:56:26.960285 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944632 5902 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14568h56m8.289312658s" Oct 30 02:56:26.960676 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.944836 5902 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:56:26.960676 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.946804 5902 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 02:56:26.960676 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.948586 5902 log.go:25] "Validated CRI v1 runtime API" Oct 30 02:56:26.960676 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.950709 5902 log.go:25] "Validated CRI v1 image API" Oct 30 02:56:26.960676 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.951508 5902 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 02:56:26.960676 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.954098 5902 fs.go:135] Filesystem UUIDs: map[520720d4-3f43-488b-bc2c-8215a6cfaee1:/dev/nvme0n1p4 549e799a-b878-462d-8408-421e1b5b776e:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 02:56:26.960917 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.954117 5902 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2/userdata/shm major:0 minor:272 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8/userdata/shm major:0 minor:337 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601/userdata/shm major:0 minor:437 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2/userdata/shm major:0 minor:439 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/79d4742442e84fb605bd48e0ff5b343cf44cbbb789374e473b6f346ac475d7cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/79d4742442e84fb605bd48e0ff5b343cf44cbbb789374e473b6f346ac475d7cf/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800/userdata/shm major:0 minor:253 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793/userdata/shm major:0 minor:372 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201/userdata/shm major:0 minor:291 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31/userdata/shm major:0 minor:435 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8/userdata/shm major:0 minor:421 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e345b3a0960da3a4969a9f4c307169a770db0fe50b76f505c5b8dd3210ef79d1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e345b3a0960da3a4969a9f4c307169a770db0fe50b76f505c5b8dd3210ef79d1/userdata/shm major:0 minor:450 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d/userdata/shm major:0 minor:384 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02ef4e11-7e6a-49ad-a017-d51735f3b124/volumes/kubernetes.io~projected/kube-api-access-b9jbh:{mountpoint:/var/lib/kubelet/pods/02ef4e11-7e6a-49ad-a017-d51735f3b124/volumes/kubernetes.io~projected/kube-api-access-b9jbh major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ab8f2d0-573d-495b-9075-bced70a5f27b/volumes/kubernetes.io~projected/kube-api-access-4pzhw:{mountpoint:/var/lib/kubelet/pods/0ab8f2d0-573d-495b-9075-bced70a5f27b/volumes/kubernetes.io~projected/kube-api-access-4pzhw major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ab8f2d0-573d-495b-9075-bced70a5f27b/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/0ab8f2d0-573d-495b-9075-bced70a5f27b/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/volumes/kubernetes.io~projected/kube-api-access-vjdmz:{mountpoint:/var/lib/kubelet/pods/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/volumes/kubernetes.io~projected/kube-api-access-vjdmz major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23217ed3-9c56-47de-be14-b34a31cc29e2/volumes/kubernetes.io~projected/kube-api-access-cgnsr:{mountpoint:/var/lib/kubelet/pods/23217ed3-9c56-47de-be14-b34a31cc29e2/volumes/kubernetes.io~projected/kube-api-access-cgnsr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b/volumes/kubernetes.io~projected/kube-api-access-p5zqc:{mountpoint:/var/lib/kubelet/pods/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b/volumes/kubernetes.io~projected/kube-api-access-p5zqc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~projected/kube-api-access-w9khp:{mountpoint:/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~projected/kube-api-access-w9khp major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/38027d42-0deb-420c-a777-546f37f24db9/volumes/kubernetes.io~projected/kube-api-access-kkhbh:{mountpoint:/var/lib/kubelet/pods/38027d42-0deb-420c-a777-546f37f24db9/volumes/kubernetes.io~projected/kube-api-access-kkhbh major:0 minor:434 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~projected/kube-api-access-p4l8l:{mountpoint:/var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~projected/kube-api-access-p4l8l major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b610fff-d6f5-4856-a97c-04893bee2bb5/volumes/kubernetes.io~projected/kube-api-access-jvhbk:{mountpoint:/var/lib/kubelet/pods/5b610fff-d6f5-4856-a97c-04893bee2bb5/volumes/kubernetes.io~projected/kube-api-access-jvhbk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~projected/kube-api-access-t2zzn:{mountpoint:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~projected/kube-api-access-t2zzn major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:368 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~projected/kube-api-access-g28pf:{mountpoint:/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~projected/kube-api-access-g28pf major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~projected/kube-api-access-d8wsw:{mountpoint:/var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~projected/kube-api-access-d8wsw major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9c32c037-b7a6-4a81-9ce9-478c53e023dd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/9c32c037-b7a6-4a81-9ce9-478c53e023dd/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf9c99a-3031-4ce6-b564-df6a1feda93c/volumes/kubernetes.io~projected/kube-api-access-cpbxl:{mountpoint:/var/lib/kubelet/pods/9cf9c99a-3031-4ce6-b564-df6a1feda93c/volumes/kubernetes.io~projected/kube-api-access-cpbxl major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~projected/kube-api-access-7d48d:{mountpoint:/var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~projected/kube-api-access-7d48d major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volumes/kubernetes.io~projected/kube-api-access-g8mqw:{mountpoint:/var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volumes/kubernetes.io~projected/kube-api-access-g8mqw major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c117c1fb-3ed4-4405-80a7-2fc1613bb98f/volumes/kubernetes.io~projected/kube-api-access-lll85:{mountpoint:/var/lib/kubelet/pods/c117c1fb-3ed4-4405-80a7-2fc1613bb98f/volumes/kubernetes.io~projected/kube-api-access-lll85 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~projected/kube-api-access-t2wnz:{mountpoint:/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~projected/kube-api-access-t2wnz major:0 minor:449 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:447 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c663f5b1-25fa-4b8a-8dc7-95f273fecd53/volumes/kubernetes.io~projected/kube-api-access-r2lxb:{mountpoint:/var/lib/kubelet/pods/c663f5b1-25fa-4b8a-8dc7-95f273fecd53/volumes/kubernetes.io~projected/kube-api-access-r2lxb major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c663f5b1-25fa-4b8a-8dc7-95f273fecd53/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c663f5b1-25fa-4b8a-8dc7-95f273fecd53/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/alertmanager/9 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/config-reloader/6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~empty-dir/config-out major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~projected/kube-api-access-cchsn:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~projected/kube-api-access-cchsn major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~projected/tls-assets major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/config-volume major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/web-config major:0 minor:303 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~projected/kube-api-access-9jv57:{mountpoint:/var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~projected/kube-api-access-9jv57 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/config-reloader/2 major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/prometheus/15 major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:396 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~projected/kube-api-access-gn2f4:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~projected/kube-api-access-gn2f4 major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~projected/tls-assets major:0 minor:407 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/config major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:401 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:404 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:403 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:409 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:408 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:402 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:405 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/web-config major:0 minor:410 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec/volumes/kubernetes.io~projected/kube-api-access-wqhsc:{mountpoint:/var/lib/kubelet/pods/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec/volumes/kubernetes.io~projected/kube-api-access-wqhsc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec/volumes/kubernetes.io~secret/metrics-certs major:0 minor:432 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/volumes/kubernetes.io~projected/kube-api-access-5nmtf:{mountpoint:/var/lib/kubelet/pods/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/volumes/kubernetes.io~projected/kube-api-access-5nmtf major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/feaaf2fd-3a0d-46a8-9252-9ea1909150df/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/feaaf2fd-3a0d-46a8-9252-9ea1909150df/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:433 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/9e3558652a09931d481b6377c1873573de26c6bdb1829bbbf13fb7e702a5553f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/0a67516bb82c79a216122e56ea97ace5f44514cde34b8b8d6169425e0ba94fda/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0ef671b46d8f53e82b03b4a8824090ce1a75f7e517a1abb46158f92b0dd0da40/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/2193c39797b17f235b414c26481ce155ff70f223015ea1d07c47093560a39f75/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/f78caee799ba8386a0e3bbe4f20d668ea8638ee23f8a3eb32f82746efe827bc8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c5fad4e463004e90b262d0e397519db854880be1fb6f2b5c5eb2637dd1ecbdc1/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/74d9f7ca29b3adcd7af88e625ee84635520d3ab1f82df7e893ba59e6dca5904e/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/6b838b1ca5c060585cd2350a8c75dd3c7c4025f06a0dcd962f302741303dc344/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-129:{mountpoint:/var/lib/containers/storage/overlay/e37efe72c6d7163232da816aba9d828e2a96c2e5318a6f66ca651a908bcf545a/merged major:0 minor:129 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/489cdab1565bb13c425b2703dbaa4a4a1f8fb5200a26a9aea5e7c8be56d51856/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/a68c7e53ea54d5544cf313f51b546d4ee8eae8dc75100e0279d8c7e9176d9903/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/56e42966cc5ccfd7f648531fd7c43a739f7c906cbba3ba12528e951b06323124/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/764a258267d7ab281e8692d2a35008e72c0d14868ff800550012f8fbaf38f548/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/f1f5af0fe3c1e0c68cc64de896da6b5dc3108601d48768016a61c087f74e66d5/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d3fd80d484fe27719a28d874e8954d80daf1b052741aa279c744ba4c4563aca5/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/64f53f0f87d0ac57e3edc9bf8fa0fb208eb52ec66d604d15c3cfbb0e53b7904d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d5fac01eb5090180fa11b04c5df404581640f0029a43bd5238ad038d947224a1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/811614cff52c92013aae259ad3982c38d21d9454f085d4d7170a296345fd7be0/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c47eb4ebc0462362499065bf0761ea55a99711cd2e9d0e9f44fdcad200f3d3a1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/c7da6cb3d2fa5bb2244eb35d7be71f10f1ff4def79106bb2d0f8c2db51669845/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/3c7ddf789f9ba9a9cf7e654e630c09bc2c41df0f835e555628385d9b032d0c0b/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f044d1c6f5fcb957b792e611c4f235a8c865ddb753051302c08df274cad16ef9/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/2d8549af9e086d6cf21a4d23ba871275213f4c27616de737a4885e0cf1ba989d/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/008b9acf5575480131b84337904e162748eef546a39a48426913a45c6965e19c/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/f0d06c4a16c2626f6f313103b1d0a399f59c5c1afe131d4ec46fa570ac512998/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/7d4c3c88858bf9436db5b940fd476a1e0678ff7dfb179bcd1af76a3f524dfa35/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b4bf29f4056402ba67b6908ab98d776deadd4918f9577935d82cb2cae8b4930c/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/b108702afeb820f373b2d3a9e5b8ed5c8d3a10b30d0beb5119efddd826f78c26/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/4524ab7475391b325c809bd28c5af9c77f8b69dcdbcceba1914082a897d13aef/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/d7e0f53e4e22111c1f8b653243d7be3a784e05c02a6afed7fa37507127f545ea/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/94e07b95c0767aa39b44638c13372510975ac5e012d228c4ba54f951b7dceb31/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/3b54c64ade468911b666afcba071c982400473b21f4a81e4f44ccfc5ae52ac3b/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/3ddb03ae11162bd45c8428369e2fbb08d2d06bf2edb52c119acfa00eec389690/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/49f41054a63ad1abe1bb6ee9c4041371f28c418aadec6d0ddb4bf7dd34ae707b/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/62af79fd7a881a33fde355aa881dc7f943ab9e600cf22194449ab6c336ea6bf0/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/c65d5571581e6084009e5017dddc343359059c8b7a53b6552de03e1968c8891a/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/6cae154cef98c5ed14e7d1a83a2489c51a60e94a7f866c49de2ec09df6b6498d/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/841de2cce55e9bac282822fd64dfb78b6020b705a7f6a16b77dbe9b68416d9c5/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/01ada936f6a6db272320e1cdcfd63b24123f10f1d3dda9ff14194eb812633f19/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/ddd005f4f44ff78c37d9ab7fd0062121ffe22dfebb429331b0cd04fec28209a3/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/e0dca5abdbf1ac84a7629b0044ef83e1a1158f9325f2847fb3ae29f99a98b2ee/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/764c2edc159bc044ce6fc6be99f4435edf3ede6a4688daf1df051c28f3501aa6/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/facb7e354bdfdf6ea86af438dcf56fb9450659a516452c66dd89b3858de2bca2/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/32e0de90c0177c53e62bf3e4ded9c7c9bc85758977a12ca4689f70b27c624943/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/d21199f7a4dddba0f8dd3dc6763c7bcda7685cc8c40abb26d0a94ee71df7b93f/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/7ccf5eca5325c81c2bf14d8d08a539ccf0afca9c01ac578122381d5ee133be68/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/a86753cd4024ab432fd82e9fb9a3bd46546d58523e7f193a81c3edf252d28c01/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/b1286079172d79b076dcc4656faf3607e41ab90e897a4a82f7673885e5367aaf/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/79ad56a84dc3e8a3962e574f568e928ef6d72cbbcd84fb849efd769f5b4b25c2/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/af8e14219d3ca4fd45c9b7383ebcedabdd17bfb5870f987ca61ab2f3d9cb9032/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/c5cfa9a1a14cdfa180d0e54b9bc69ed06ed24acafd390229d566c7e82e62226f/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/4284a65b780f02dc6341e6a1c69e41576f2416fcf942f5be93589f6c96b83c6e/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-388:{mountpoint:/var/lib/containers/storage/overlay/5060ad90297cb439964cd9446dcddb33231fc654c71fbf3d1891263e0859a157/merged major:0 minor:388 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/e700d13181bdb4a0a2159b0b544ccbbaf602e916e16b161f6ffe2aee0d5495ac/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/9eef4aaed690528ffcb12aafde25bbf5805d979d5eeeea161ebdeaaffcb5faee/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/66e8479959d12e752becc117359bca7a799ca368e5a4fbe286caa23a7722c0e8/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/885eca2af19db0f3b0ebf5b03055e57313d602fdc986fcfc93b3a03e36260fdb/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-427:{mountpoint:/var/lib/containers/storage/overlay/602e29cd6f8766fb8fd27c30eeeaf97d458b78c686c95e1923f5ed6d02e53dde/merged major:0 minor:427 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/b69e109a779a024b75c205f12b4af0a1719cdd15afe695dfd243335320b61aac/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-443:{mountpoint:/var/lib/containers/storage/overlay/1fcab7282270d794b3b9f6938006dcaf1a9ee7770b009c874bd418536553590a/merged major:0 minor:443 fsType:overlay blockSize:0} overlay_0-445:{mountpoint:/var/lib/containers/storage/overlay/686ce7bed9ff02df9037951d3fcb4bbcf04f5603cc0ae56077294ff531d80210/merged major:0 minor:445 fsType:overlay blockSize:0} overlay_0-452:{mountpoint:/var/lib/containers/storage/overlay/f250ceda519c077fa5b9000811d13390eceec7b861158e85616a617dc6c2c556/merged major:0 minor:452 fsType:overlay blockSize:0} overlay_0-454:{mountpoint:/var/lib/containers/storage/overlay/74611270c09ae5fed06f708a0bed0a56035a9d20ce7d4d91262e825fa369ac06/merged major:0 minor:454 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/1862c44d8c94803c1f8689304b22445e6a4adeacc332f8ee2a719bfe5a5adcfd/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-462:{mountpoint:/var/lib/containers/storage/overlay/2ad5468a48eeeffb5691016cefe5ebc2cb449129695a5f4b5d27160715705a0a/merged major:0 minor:462 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/792441f784d86e91e4bc5e15cbdef342025c6e9ddb40f504fa6f05aa8052ec9c/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-473:{mountpoint:/var/lib/containers/storage/overlay/de3e4b2efa4715d31abd6e04971af26c420b160e999529ff69d3c5d126f330c9/merged major:0 minor:473 fsType:overlay blockSize:0} overlay_0-479:{mountpoint:/var/lib/containers/storage/overlay/8af40b7e4b62d7832ac508f4da1f770e45994488bf1e76842073aa21c6ce6ec6/merged major:0 minor:479 fsType:overlay blockSize:0} overlay_0-484:{mountpoint:/var/lib/containers/storage/overlay/c04911d23d05e05e02d6f51993d2b77a3566b8d9002eb289f909655ac2e6ac05/merged major:0 minor:484 fsType:overlay blockSize:0} overlay_0-490:{mountpoint:/var/lib/containers/storage/overlay/51f8c00076d11a14d9dd1e7b236c5db95b639d7284306021cdb0fb15fb46930e/merged major:0 minor:490 fsType:overlay blockSize:0} overlay_0-495:{mountpoint:/var/lib/containers/storage/overlay/c8c88f52889900261a293c6119ca312f746b1e621759620fd478b89194c2d8e4/merged major:0 minor:495 fsType:overlay blockSize:0} overlay_0-500:{mountpoint:/var/lib/containers/storage/overlay/ee5c62c70e00e773780ff7768ca2aeb95fc1bf4637272d761b521dc53dde2916/merged major:0 minor:500 fsType:overlay blockSize:0} overlay_0-505:{mountpoint:/var/lib/containers/storage/overlay/36e6ec81817eada6648b8a3a6e2e8645b37d9c6ce04b7601f0718452f9eb5b28/merged major:0 minor:505 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/500268cebdf22e506490ec626051e65815cf3fd806c03339d96701c7e1bd48bb/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/dc26674a8e80d7bb055b99b4ffeae78896a44c114773533d1abbd581c6def00d/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/4b71bec3104026dfd8af1f799fd5d08ce2db74384a8d5a44a28727670cb37392/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/b397770281c46a0c9f6382de760b88a229b0622e7397b8d4ceb4ddb3e8ae2a8a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/85fc76762b21d3d80912a01bdddf215ec9393060ef48289bc1e5f3ec798d9950/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/1ea7afd4f2c000882590cee8f87c8d674da8b1f06cfe0cf1faf823165956b4b7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1a1d91630b895ffec6e7d6dfb95222c927734d938d07691d5823474b90e9fcc7/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/2634b324b0c7e875a8d344811e8e32eb1b0bb76758cfd6b433eb8f244e8d45e0/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5db71c3ff66aafdebae46511beea25bef2e7c3571c4bf5c948ea04b3f242b6e1/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/f024b48d67feeab22d0e63853df13b93c056c36a44df0c6a17d636ace021d176/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 02:56:26.964687 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.964174 5902 manager.go:217] Machine: {Timestamp:2026-10-30 02:56:26.963690485 +0000 UTC m=+0.061107036 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099246 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec5f04c246e0490ccaa23ac758501 SystemUUID:ec2ec5f0-4c24-6e04-90cc-aa23ac758501 BootID:287fe92e-ebd3-4185-80f0-6a709581f4e1 Filesystems:[{Device:overlay_0-129 DeviceMajor:0 DeviceMinor:129 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:368 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:405 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~projected/kube-api-access-gn2f4 DeviceMajor:0 DeviceMinor:411 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:447 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:299 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:432 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~projected/kube-api-access-t2wnz DeviceMajor:0 DeviceMinor:449 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:298 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:301 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/520947f4e2382e548b764c34c85966e69e9455570fc2300b299e3169a5dce1f2/userdata/shm DeviceMajor:0 DeviceMinor:439 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-473 DeviceMajor:0 DeviceMinor:473 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-490 DeviceMajor:0 DeviceMinor:490 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e87d3b31ffbc1b80704a8374b378bb7637168b795d8cb49b2cac4a4aebc1a5fe/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/18af35b7be94d88aa423b1edf20a600d8f5d2f6f5c864b6175a14414b19af5f9/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-388 DeviceMajor:0 DeviceMinor:388 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f6bf4194308425049b1d08dd6e3eaa4393c287ec740bee498ac2a548ac496e51/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d/userdata/shm DeviceMajor:0 DeviceMinor:384 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~projected/kube-api-access-7d48d DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/caae5b2393ce673c09b30e8d2b3e285e3de4d6604ffb9263fc5242d4e5179445/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~projected/kube-api-access-w9khp DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9c32c037-b7a6-4a81-9ce9-478c53e023dd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/634712d39c20210d5d5aa30841cdb1998c8e2fe1bd9023667f4192bb5be06992/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:448 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-452 DeviceMajor:0 DeviceMinor:452 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-454 DeviceMajor:0 DeviceMinor:454 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volumes/kubernetes.io~projected/kube-api-access-g8mqw DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c117c1fb-3ed4-4405-80a7-2fc1613bb98f/volumes/kubernetes.io~projected/kube-api-access-lll85 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/volumes/kubernetes.io~projected/kube-api-access-5nmtf DeviceMajor:0 DeviceMinor:336 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:406 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa210095be0d028422dde2dfb027b859cf0b4269cbcd34728ee9834474a73128/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~projected/kube-api-access-p4l8l DeviceMajor:0 DeviceMinor:252 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:364 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-462 DeviceMajor:0 DeviceMinor:462 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b0c909b85f8265281ff90180d98d32c2962a62b2e9218a844e3545f5d5f68ad/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:302 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~projected/kube-api-access-cchsn DeviceMajor:0 DeviceMinor:304 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-427 DeviceMajor:0 DeviceMinor:427 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b/volumes/kubernetes.io~projected/kube-api-access-p5zqc DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c663f5b1-25fa-4b8a-8dc7-95f273fecd53/volumes/kubernetes.io~projected/kube-api-access-r2lxb DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-443 DeviceMajor:0 DeviceMinor:443 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b610fff-d6f5-4856-a97c-04893bee2bb5/volumes/kubernetes.io~projected/kube-api-access-jvhbk DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf9c99a-3031-4ce6-b564-df6a1feda93c/volumes/kubernetes.io~projected/kube-api-access-cpbxl DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9b402df1dbd37b53f0909a2f3f73000cb574e30666562151596e3653de413201/userdata/shm DeviceMajor:0 DeviceMinor:291 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~projected/kube-api-access-t2zzn DeviceMajor:0 DeviceMinor:371 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:401 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0ab8f2d0-573d-495b-9075-bced70a5f27b/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c663f5b1-25fa-4b8a-8dc7-95f273fecd53/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:251 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~projected/kube-api-access-d8wsw DeviceMajor:0 DeviceMinor:285 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/16c0fd65cedfebdb7da636d3a68618e7be997fd0d92246addd48fbbc7f4b9dc8/userdata/shm DeviceMajor:0 DeviceMinor:337 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/79d4742442e84fb605bd48e0ff5b343cf44cbbb789374e473b6f346ac475d7cf/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:257 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1f96b77f37a6245d0714fac2e0e32625cdc508a8f9e80e6554d09e8d6521034/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~projected/kube-api-access-g28pf DeviceMajor:0 DeviceMinor:261 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:258 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-495 DeviceMajor:0 DeviceMinor:495 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0ab8f2d0-573d-495b-9075-bced70a5f27b/volumes/kubernetes.io~projected/kube-api-access-4pzhw DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2/userdata/shm DeviceMajor:0 DeviceMinor:272 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:369 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/62a522bc-31b4-477f-8a93-4d8959c0aabf/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:370 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8ace187de3cf9255b76756b2b1b2a108e5bbc64d60eb928c95074e1368def793/userdata/shm DeviceMajor:0 DeviceMinor:372 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-484 DeviceMajor:0 DeviceMinor:484 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b2ca4fd079285cc77e0f47f8737aa0bb77bf3b2683fc81850428a7fbaff01d2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8/userdata/shm DeviceMajor:0 DeviceMinor:421 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/feaaf2fd-3a0d-46a8-9252-9ea1909150df/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:433 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e345b3a0960da3a4969a9f4c307169a770db0fe50b76f505c5b8dd3210ef79d1/userdata/shm DeviceMajor:0 DeviceMinor:450 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-505 DeviceMajor:0 DeviceMinor:505 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/23217ed3-9c56-47de-be14-b34a31cc29e2/volumes/kubernetes.io~projected/kube-api-access-cgnsr DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7683078059158cf87581a899858adf9eb657800ef868944130321c3f4ebc85f/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:283 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:378 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a577fb95-15eb-4954-91f1-426fca88fb5b/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/volumes/kubernetes.io~projected/kube-api-access-vjdmz DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e6bca31ff36afe80224ca23909424d5ae871109c539e7b41517df78a4d7a800/userdata/shm DeviceMajor:0 DeviceMinor:253 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/volumes/kubernetes.io~projected/kube-api-access-9jv57 DeviceMajor:0 DeviceMinor:286 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:404 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:408 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:410 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a9b5ab42-4a77-496d-bb0b-b1dd59680622/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:403 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:409 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-445 DeviceMajor:0 DeviceMinor:445 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-479 DeviceMajor:0 DeviceMinor:479 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85d227a6-403a-4b90-9750-eb26433e0758/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:282 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:296 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:297 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac0d52cdd2e710249e8a3229033e52cacf80eac3dbd1abb5f480612a9f4f1e31/userdata/shm DeviceMajor:0 DeviceMinor:435 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-500 DeviceMajor:0 DeviceMinor:500 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/02ef4e11-7e6a-49ad-a017-d51735f3b124/volumes/kubernetes.io~projected/kube-api-access-b9jbh DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4259b75f-532b-4e73-9d4e-0d3d72909feb/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:332 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:303 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:396 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:402 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:407 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6a43a784e0a761a7c4704ad9f56629aad76b0cd9573d563e1d7ee96824aae75f/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:300 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/38027d42-0deb-420c-a777-546f37f24db9/volumes/kubernetes.io~projected/kube-api-access-kkhbh DeviceMajor:0 DeviceMinor:434 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23a372bcc84ab9aa60f0bc3a761c1765d14acbba1db767bc206c11fc6001a601/userdata/shm DeviceMajor:0 DeviceMinor:437 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec/volumes/kubernetes.io~projected/kube-api-access-wqhsc DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0ae90a8754e1b82 MacAddress:0e:7b:10:58:e6:5d Speed:10000 Mtu:8901} {Name:16c0fd65cedfebd MacAddress:02:b1:cb:ff:01:c2 Speed:10000 Mtu:8901} {Name:23a372bcc84ab9a MacAddress:3e:b8:0c:b4:23:34 Speed:10000 Mtu:8901} {Name:520947f4e2382e5 MacAddress:1e:8c:9a:6d:e6:5a Speed:10000 Mtu:8901} {Name:6b5f8544533266e MacAddress:ba:7b:20:1f:2d:63 Speed:10000 Mtu:8901} {Name:79d4742442e84fb MacAddress:fa:51:18:79:34:3d Speed:10000 Mtu:8901} {Name:7e6bca31ff36afe MacAddress:fa:d3:06:47:ac:53 Speed:10000 Mtu:8901} {Name:8ace187de3cf925 MacAddress:16:8f:21:6d:99:91 Speed:10000 Mtu:8901} {Name:9b402df1dbd37b5 MacAddress:ae:96:a1:c9:10:b1 Speed:10000 Mtu:8901} {Name:ac0d52cdd2e7102 MacAddress:92:d8:44:8c:3e:ac Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:cf:38:f5:c3 Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:1f:1b:58:fc:6d Speed:0 Mtu:8901} {Name:d3c78350571a943 MacAddress:e6:99:4e:9d:1c:19 Speed:10000 Mtu:8901} {Name:e345b3a0960da3a MacAddress:42:35:d8:26:d1:d7 Speed:10000 Mtu:8901} {Name:e87d3b31ffbc1b8 MacAddress:66:77:bd:6d:38:80 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:cf:38:f5:c3 Speed:0 Mtu:9001} {Name:f6bf41943084250 MacAddress:d2:cf:79:e9:5d:37 Speed:10000 Mtu:8901} {Name:f75dbbb006632ff MacAddress:7e:bd:b6:b2:e5:a9 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:e6:49:1a:c2:b5:14 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:aa:56:82:f3:64:75 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 02:56:26.965168 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.964692 5902 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 02:56:26.965168 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.964750 5902 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 02:56:26.965275 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965217 5902 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 02:56:26.965438 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965250 5902 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-236.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 02:56:26.965438 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965440 5902 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 02:56:26.965573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965450 5902 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 02:56:26.965573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965465 5902 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:56:26.965573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965485 5902 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 02:56:26.965663 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965587 5902 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 02:56:26.965792 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965782 5902 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 02:56:26.965928 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965919 5902 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 02:56:26.965961 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965954 5902 kubelet.go:493] "Attempting to sync node with API server" Oct 30 02:56:26.966036 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965965 5902 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 02:56:26.966036 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965976 5902 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 02:56:26.966036 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965985 5902 kubelet.go:405] "Adding apiserver pod source" Oct 30 02:56:26.966036 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.965992 5902 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 02:56:26.967058 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.967040 5902 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 02:56:26.967058 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.967059 5902 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:56:26.967902 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.967876 5902 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 02:56:26.972158 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972141 5902 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 02:56:26.972158 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972156 5902 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 02:56:26.972278 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972266 5902 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 02:56:26.972503 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972492 5902 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 02:56:26.972533 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972522 5902 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 02:56:26.972658 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972649 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 02:56:26.972688 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972665 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 02:56:26.972688 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972671 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 02:56:26.972688 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972677 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 02:56:26.972688 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972683 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972691 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972701 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972708 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972715 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972721 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972729 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972739 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972765 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 02:56:26.972804 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.972773 5902 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 02:56:26.976265 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.976251 5902 server.go:1263] "Started kubelet" Oct 30 02:56:26.976683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.976325 5902 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 02:56:26.976683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.976349 5902 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 02:56:26.976683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.976435 5902 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 02:56:26.976933 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.976775 5902 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 02:56:26.976831 ip-10-0-142-236 systemd[1]: Started Kubernetes Kubelet. Oct 30 02:56:26.977595 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.977532 5902 server.go:317] "Adding debug handlers to kubelet server" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.980473 5902 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.980500 5902 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.980586 5902 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.980601 5902 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:26.980606 5902 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-142-236.ec2.internal\" not found" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.980715 5902 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.981311 5902 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 02:50:16 +0000 UTC" deadline="2028-05-07 05:13:21.342557446 +0000 UTC" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.981356 5902 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13322h16m54.361204886s" Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.981502 5902 factory.go:55] Registering systemd factory Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.981526 5902 factory.go:223] Registration of the systemd container factory successfully Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.982656 5902 factory.go:153] Registering CRI-O factory Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.982673 5902 factory.go:223] Registration of the crio container factory successfully Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.982737 5902 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.982756 5902 factory.go:103] Registering Raw factory Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.982773 5902 manager.go:1199] Started watching for new ooms in manager Oct 30 02:56:26.984150 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.983565 5902 manager.go:319] Starting recovery of all containers Oct 30 02:56:26.989957 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.989913 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 02:56:26.990046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.989960 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb" volumeName="kubernetes.io/configmap/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 02:56:26.990046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.989981 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert" seLinuxMountContext="" Oct 30 02:56:26.990046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.989996 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d227a6-403a-4b90-9750-eb26433e0758" volumeName="kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 02:56:26.990046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990011 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5ab42-4a77-496d-bb0b-b1dd59680622" volumeName="kubernetes.io/projected/a9b5ab42-4a77-496d-bb0b-b1dd59680622-kube-api-access-g8mqw" seLinuxMountContext="" Oct 30 02:56:26.990046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990029 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.990046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990042 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990060 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d227a6-403a-4b90-9750-eb26433e0758" volumeName="kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-metrics-client-ca" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990077 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990093 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990106 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab8f2d0-573d-495b-9075-bced70a5f27b" volumeName="kubernetes.io/secret/0ab8f2d0-573d-495b-9075-bced70a5f27b-metrics-tls" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990122 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990142 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990154 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990178 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990190 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config" seLinuxMountContext="" Oct 30 02:56:26.990234 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990208 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990238 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/projected/62a522bc-31b4-477f-8a93-4d8959c0aabf-kube-api-access-t2zzn" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990260 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c663f5b1-25fa-4b8a-8dc7-95f273fecd53" volumeName="kubernetes.io/secret/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-cert" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990278 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990291 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="38027d42-0deb-420c-a777-546f37f24db9" volumeName="kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990309 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990323 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990339 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990352 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ef4e11-7e6a-49ad-a017-d51735f3b124" volumeName="kubernetes.io/empty-dir/02ef4e11-7e6a-49ad-a017-d51735f3b124-tmp-dir" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990364 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c663f5b1-25fa-4b8a-8dc7-95f273fecd53" volumeName="kubernetes.io/projected/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-kube-api-access-r2lxb" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990381 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990393 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02ef4e11-7e6a-49ad-a017-d51735f3b124" volumeName="kubernetes.io/projected/02ef4e11-7e6a-49ad-a017-d51735f3b124-kube-api-access-b9jbh" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990418 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4259b75f-532b-4e73-9d4e-0d3d72909feb" volumeName="kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990432 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990449 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf9c99a-3031-4ce6-b564-df6a1feda93c" volumeName="kubernetes.io/configmap/9cf9c99a-3031-4ce6-b564-df6a1feda93c-iptables-alerter-script" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990461 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config" seLinuxMountContext="" Oct 30 02:56:26.990487 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990483 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990497 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-client-tls" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990510 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990535 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c32c037-b7a6-4a81-9ce9-478c53e023dd" volumeName="kubernetes.io/configmap/9c32c037-b7a6-4a81-9ce9-478c53e023dd-konnectivity-ca" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990547 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990562 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990575 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990587 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d227a6-403a-4b90-9750-eb26433e0758" volumeName="kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990604 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990617 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" volumeName="kubernetes.io/projected/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-kube-api-access-9jv57" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990635 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23217ed3-9c56-47de-be14-b34a31cc29e2" volumeName="kubernetes.io/configmap/23217ed3-9c56-47de-be14-b34a31cc29e2-serviceca" seLinuxMountContext="" Oct 30 02:56:26.990938 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990648 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-metrics-client-ca" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.990991 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" volumeName="kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-metrics-client-ca" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991029 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991043 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf9c99a-3031-4ce6-b564-df6a1feda93c" volumeName="kubernetes.io/projected/9cf9c99a-3031-4ce6-b564-df6a1feda93c-kube-api-access-cpbxl" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991056 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" volumeName="kubernetes.io/projected/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-kube-api-access-wqhsc" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991068 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a577fb95-15eb-4954-91f1-426fca88fb5b" volumeName="kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-tuned" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991079 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5ab42-4a77-496d-bb0b-b1dd59680622" volumeName="kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991101 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c117c1fb-3ed4-4405-80a7-2fc1613bb98f" volumeName="kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cni-binary-copy" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991112 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991123 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab8f2d0-573d-495b-9075-bced70a5f27b" volumeName="kubernetes.io/empty-dir/0ab8f2d0-573d-495b-9075-bced70a5f27b-tmp-dir" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991134 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991145 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991156 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b610fff-d6f5-4856-a97c-04893bee2bb5" volumeName="kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991179 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991189 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9" volumeName="kubernetes.io/projected/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-kube-api-access-5nmtf" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991204 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991215 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991226 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" volumeName="kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991238 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab8f2d0-573d-495b-9075-bced70a5f27b" volumeName="kubernetes.io/configmap/0ab8f2d0-573d-495b-9075-bced70a5f27b-config-volume" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991252 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991262 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4259b75f-532b-4e73-9d4e-0d3d72909feb" volumeName="kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-tls" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991274 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c117c1fb-3ed4-4405-80a7-2fc1613bb98f" volumeName="kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-daemon-config" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991286 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991300 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991314 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991333 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d227a6-403a-4b90-9750-eb26433e0758" volumeName="kubernetes.io/projected/85d227a6-403a-4b90-9750-eb26433e0758-kube-api-access-d8wsw" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991346 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991360 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991373 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb" volumeName="kubernetes.io/empty-dir/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-data-volume" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991401 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb" volumeName="kubernetes.io/projected/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-api-access-vjdmz" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991414 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991427 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b610fff-d6f5-4856-a97c-04893bee2bb5" volumeName="kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 02:56:26.991423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991438 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-trusted-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991449 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991461 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" volumeName="kubernetes.io/empty-dir/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-textfile" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991492 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4259b75f-532b-4e73-9d4e-0d3d72909feb" volumeName="kubernetes.io/projected/4259b75f-532b-4e73-9d4e-0d3d72909feb-kube-api-access-p4l8l" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991504 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-federate-client-tls" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991516 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991528 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b610fff-d6f5-4856-a97c-04893bee2bb5" volumeName="kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-binary-copy" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991552 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991564 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991577 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23217ed3-9c56-47de-be14-b34a31cc29e2" volumeName="kubernetes.io/projected/23217ed3-9c56-47de-be14-b34a31cc29e2-kube-api-access-cgnsr" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991601 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b610fff-d6f5-4856-a97c-04893bee2bb5" volumeName="kubernetes.io/projected/5b610fff-d6f5-4856-a97c-04893bee2bb5-kube-api-access-jvhbk" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991615 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991627 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="68dc07a5-b6fe-4cac-a23b-e753b406161c" volumeName="kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991640 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c117c1fb-3ed4-4405-80a7-2fc1613bb98f" volumeName="kubernetes.io/projected/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-kube-api-access-lll85" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991653 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991665 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991677 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62a522bc-31b4-477f-8a93-4d8959c0aabf" volumeName="kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991691 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a577fb95-15eb-4954-91f1-426fca88fb5b" volumeName="kubernetes.io/projected/a577fb95-15eb-4954-91f1-426fca88fb5b-kube-api-access-7d48d" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991701 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5ab42-4a77-496d-bb0b-b1dd59680622" volumeName="kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-env-overrides" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991724 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991739 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991752 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="feaaf2fd-3a0d-46a8-9252-9ea1909150df" volumeName="kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991765 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a5034dc-e3d7-4af9-a808-d2ffad6ba46b" volumeName="kubernetes.io/projected/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kube-api-access-p5zqc" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991779 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5ab42-4a77-496d-bb0b-b1dd59680622" volumeName="kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-script-lib" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991801 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a9b5ab42-4a77-496d-bb0b-b1dd59680622" volumeName="kubernetes.io/secret/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991829 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991844 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eabb308d-a5a1-4207-bc27-c81e6bb4b2ec" volumeName="kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991856 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" volumeName="kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991868 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991882 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d227a6-403a-4b90-9750-eb26433e0758" volumeName="kubernetes.io/empty-dir/85d227a6-403a-4b90-9750-eb26433e0758-volume-directive-shadow" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991895 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991909 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4259b75f-532b-4e73-9d4e-0d3d72909feb" volumeName="kubernetes.io/configmap/4259b75f-532b-4e73-9d4e-0d3d72909feb-metrics-client-ca" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991922 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9c32c037-b7a6-4a81-9ce9-478c53e023dd" volumeName="kubernetes.io/secret/9c32c037-b7a6-4a81-9ce9-478c53e023dd-agent-certs" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991933 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a577fb95-15eb-4954-91f1-426fca88fb5b" volumeName="kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-tmp" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991946 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991961 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" volumeName="kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.991973 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb" volumeName="kubernetes.io/secret/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992002 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2a50d657-6d01-4305-b6f5-06eeaa081027" volumeName="kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992015 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85d227a6-403a-4b90-9750-eb26433e0758" volumeName="kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992027 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992037 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9" volumeName="kubernetes.io/secret/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992047 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0ab8f2d0-573d-495b-9075-bced70a5f27b" volumeName="kubernetes.io/projected/0ab8f2d0-573d-495b-9075-bced70a5f27b-kube-api-access-4pzhw" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992054 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d172e77b-1700-49d4-8513-07b3a516b821" volumeName="kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992062 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" volumeName="kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-tls" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992070 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992078 5902 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d3601216-ec2f-45de-9685-f97ac814e4a7" volumeName="kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets" seLinuxMountContext="" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992089 5902 reconstruct.go:97] "Volume reconstruction finished" Oct 30 02:56:26.993014 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:26.992097 5902 reconciler.go:29] "Reconciler: start to sync state" Oct 30 02:56:27.011535 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.011483 5902 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 02:56:27.012967 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.012939 5902 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 02:56:27.012967 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.012962 5902 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 02:56:27.013147 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.012983 5902 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 02:56:27.013147 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:27.013022 5902 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 02:56:27.030452 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.030414 5902 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 02:56:27.032734 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.032697 5902 generic.go:356] "Generic (PLEG): container finished" podID="8c66d532a44fea0497f5e76e786c3710" containerID="54cc8fae25303f61b5953e43282c355be3a44f100a436c3174b9e55da1553ae4" exitCode=0 Oct 30 02:56:27.041324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.041299 5902 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:56:27.041476 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.041301 5902 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:56:27.047530 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.047510 5902 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:56:27.048003 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.047975 5902 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 02:56:27.048146 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.048124 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02" exitCode=0 Oct 30 02:56:27.054163 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.054135 5902 generic.go:356] "Generic (PLEG): container finished" podID="d25c69c2-ac9b-492c-9332-bd1e3abd1f1b" containerID="09f200e56c45d031680836392306e859809e33fae0a98510a19ffd52c04e4103" exitCode=0 Oct 30 02:56:27.068353 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.068323 5902 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="9bc97544f5a9f06b013ecc818b068daebb75f4f7b11779da15d8aacf44ee229e" exitCode=0 Oct 30 02:56:27.068353 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.068349 5902 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="b3979947a7f62f1b28b2d9d9f4676037e7ae7ee31e0f8e652188c7f8fda646c7" exitCode=0 Oct 30 02:56:27.068523 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.068359 5902 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="526501a468726f16618f5127787c77c54bc83cf150771d4b52d9c4fceb24b3af" exitCode=0 Oct 30 02:56:27.068523 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.068368 5902 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="d8ee55609b99c76f98dabc05a8bce1f4cd58537ac9a96d6d2ea4da196dcfa0ef" exitCode=0 Oct 30 02:56:27.068523 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.068374 5902 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="c2d61875c72c3565480d3b28ecf3021c0d1cda73eeee7ca592469c72f2127087" exitCode=0 Oct 30 02:56:27.068523 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.068379 5902 generic.go:356] "Generic (PLEG): container finished" podID="5b610fff-d6f5-4856-a97c-04893bee2bb5" containerID="130e3f3e2c4515adc239ab0bb4c2bf4a34fb394e78182c646333fa500729ea4c" exitCode=0 Oct 30 02:56:27.073744 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.073715 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="c130d96630037f1c2d0f3333672ef1308f65196d728dd99ff514a37600d042ef" exitCode=0 Oct 30 02:56:27.109083 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.109025 5902 manager.go:324] Recovery completed Oct 30 02:56:27.113305 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:27.113278 5902 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 02:56:27.134502 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134478 5902 cpu_manager.go:226] "Starting" policy="none" Oct 30 02:56:27.134502 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134494 5902 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 02:56:27.134637 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134514 5902 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 02:56:27.134668 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134644 5902 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 02:56:27.134725 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134656 5902 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 02:56:27.134725 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134676 5902 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 02:56:27.134725 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134684 5902 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 02:56:27.134725 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134689 5902 policy_none.go:50] "Start" Oct 30 02:56:27.134725 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134698 5902 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 02:56:27.134725 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134708 5902 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 02:56:27.134899 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134848 5902 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 02:56:27.134899 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134856 5902 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 02:56:27.134899 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.134862 5902 policy_none.go:44] "Start" Oct 30 02:56:27.139869 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.139835 5902 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 02:56:27.143906 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.143893 5902 manager.go:341] "Starting Device Plugin manager" Oct 30 02:56:27.144514 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:27.143920 5902 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 02:56:27.144560 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.144518 5902 server.go:87] "Starting device plugin registration server" Oct 30 02:56:27.144777 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.144767 5902 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 02:56:27.144847 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.144779 5902 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 02:56:27.144922 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.144905 5902 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 02:56:27.145045 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.145009 5902 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 02:56:27.145104 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.145063 5902 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 02:56:27.145104 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.145071 5902 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 02:56:27.145182 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.145118 5902 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T02:56:27.145045112Z","UUID":"2de7486f-6e14-4b26-825f-1ae51c37ba99","Handler":null,"Name":"","Endpoint":""} Oct 30 02:56:27.146928 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.146906 5902 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 02:56:27.147017 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.146967 5902 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 02:56:27.147170 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:27.147145 5902 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 02:56:27.245887 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.245857 5902 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 02:56:27.247858 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.247839 5902 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientMemory" Oct 30 02:56:27.247950 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.247872 5902 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 02:56:27.247950 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.247884 5902 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-142-236.ec2.internal" event="NodeHasSufficientPID" Oct 30 02:56:27.248030 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.247953 5902 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.282898 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.282879 5902 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.282949 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.282940 5902 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.313631 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.313584 5902 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal"] Oct 30 02:56:27.314308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314271 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"0c527eada420e7f4995c8eee6de2ab03","Type":"ContainerStarted","Data":"486bbd0e376b1d2bbe8ba6ea90a6c59bcf6d90a587f7860b872e08fb9afb144b"} Oct 30 02:56:27.314308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314304 5902 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.314308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314314 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" event={"ID":"0c527eada420e7f4995c8eee6de2ab03","Type":"ContainerStarted","Data":"5e0c9512aa2d4562858b7c72d9e3e9cc3c151553897504740a3e8a9012d177fd"} Oct 30 02:56:27.314561 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314334 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"8c66d532a44fea0497f5e76e786c3710","Type":"ContainerStarted","Data":"4cbc0fc332503e7a0b92995945059502f78e13a2b7056060cd98188857a234bd"} Oct 30 02:56:27.314561 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314351 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"8c66d532a44fea0497f5e76e786c3710","Type":"ContainerDied","Data":"54cc8fae25303f61b5953e43282c355be3a44f100a436c3174b9e55da1553ae4"} Oct 30 02:56:27.314561 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314370 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" event={"ID":"8c66d532a44fea0497f5e76e786c3710","Type":"ContainerStarted","Data":"7f097f440ad2cc847ce64d5a5cf195ce0c87e22778182ec75634f108d778bb68"} Oct 30 02:56:27.314561 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.314472 5902 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.347656 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.347626 5902 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:56:27.347757 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:27.347678 5902 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.349154 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.349143 5902 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:56:27.349193 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:27.349168 5902 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.431521 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.431443 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.431521 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.431474 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.431521 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.431497 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c527eada420e7f4995c8eee6de2ab03-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"0c527eada420e7f4995c8eee6de2ab03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.532029 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.531994 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.532029 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.532036 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c527eada420e7f4995c8eee6de2ab03-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"0c527eada420e7f4995c8eee6de2ab03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.532198 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.532053 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.532198 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.532101 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.532198 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.532104 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0c527eada420e7f4995c8eee6de2ab03-config\") pod \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" (UID: \"0c527eada420e7f4995c8eee6de2ab03\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.532198 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.532100 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c66d532a44fea0497f5e76e786c3710-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" (UID: \"8c66d532a44fea0497f5e76e786c3710\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:27.966888 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:27.966847 5902 apiserver.go:52] "Watching apiserver" Oct 30 02:56:28.014558 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.014526 5902 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 02:56:28.015214 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.015191 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0","openshift-multus/multus-zdmr4","openshift-console/console-5d9f9f777-vk5jc","kube-system/global-pull-secret-syncer-569p6","kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal","openshift-monitoring/monitoring-plugin-77f4f76594-j69dd","openshift-multus/multus-additional-cni-plugins-4t8gf","openshift-monitoring/node-exporter-ffn7c","openshift-console/console-58955f585-4thl7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk","openshift-ingress-canary/ingress-canary-hpxr7","openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc","openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh","kube-system/konnectivity-agent-vhplh","openshift-cluster-node-tuning-operator/tuned-nlh5q","openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f","openshift-network-diagnostics/network-check-target-knnkw","openshift-network-operator/iptables-alerter-7p5r2","openshift-console/console-7db894548b-j267d","openshift-multus/network-metrics-daemon-zmq8r","openshift-dns/dns-default-gp6sv","openshift-dns/node-resolver-pkc7w","openshift-ovn-kubernetes/ovnkube-node-lczmd","openshift-insights/insights-runtime-extractor-rflzz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal","openshift-monitoring/prometheus-k8s-0","openshift-image-registry/node-ca-fsp8w"] Oct 30 02:56:28.032651 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.032622 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 02:56:28.032901 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.032885 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-s7n2w\"" Oct 30 02:56:28.033006 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.032974 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-pchn6\"" Oct 30 02:56:28.033196 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.033181 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-65dst\"" Oct 30 02:56:28.045389 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045364 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 02:56:28.045533 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045504 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.045728 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045708 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.045790 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045735 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q5wfc\"" Oct 30 02:56:28.045790 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045742 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 02:56:28.045790 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045755 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.045790 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045768 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045791 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-vcctk\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045799 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045828 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045864 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045935 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045977 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 02:56:28.045990 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.045979 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046013 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046021 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046015 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046034 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046104 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046112 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046160 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046189 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046197 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046197 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046219 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046231 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.046301 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046293 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.046807 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046322 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.046807 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.046349 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.047324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.047302 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-5kvz4\"" Oct 30 02:56:28.047416 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.047338 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.048042 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048025 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 02:56:28.048264 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048250 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 02:56:28.048424 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048412 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5fp68\"" Oct 30 02:56:28.048518 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048506 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-726c9\"" Oct 30 02:56:28.048561 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048554 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 02:56:28.048696 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048679 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cth5s\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048682 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048703 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qr586\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048688 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048727 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048731 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048708 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048730 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lqw84\"" Oct 30 02:56:28.048761 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048757 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-q4rkf\"" Oct 30 02:56:28.049047 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048745 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 02:56:28.049047 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048884 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 02:56:28.049047 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.048924 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 02:56:28.049047 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.049030 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 02:56:28.049149 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.049071 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pz2d6\"" Oct 30 02:56:28.049149 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.049110 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lx7sd\"" Oct 30 02:56:28.049265 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.049245 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-mdrmf\"" Oct 30 02:56:28.051609 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.051584 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.051964 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.051947 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 02:56:28.054568 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.054555 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 02:56:28.055948 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.055935 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.083214 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.083191 5902 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 02:56:28.092299 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.092279 5902 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:28.092412 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.092349 5902 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:28.123503 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.123473 5902 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:56:28.123609 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:28.123521 5902 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-142-236.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-236.ec2.internal" Oct 30 02:56:28.124806 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.124791 5902 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 02:56:28.124887 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:28.124839 5902 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-236.ec2.internal" Oct 30 02:56:28.133678 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.133657 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 02:56:28.133840 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.133811 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 02:56:28.133898 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.133857 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 02:56:28.133947 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.133912 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 02:56:28.135046 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135029 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.135090 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135054 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.135090 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135073 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-run\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.135090 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135087 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.135204 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135105 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.135204 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135177 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.135343 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135328 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.135413 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135382 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8wsw\" (UniqueName: \"kubernetes.io/projected/85d227a6-403a-4b90-9750-eb26433e0758-kube-api-access-d8wsw\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.135413 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135400 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.135515 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135401 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.135515 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135427 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.135515 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135457 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:28.135515 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135482 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135510 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135540 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-federate-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135573 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-crio-socket\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135607 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-textfile\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135650 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-tls\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135673 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135671 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135702 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf9c99a-3031-4ce6-b564-df6a1feda93c-host-slash\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.135708 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135704 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-metrics-certs\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135720 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-textfile\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135745 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-etc-kubernetes\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135806 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-systemd-units\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135850 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135869 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-modprobe-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135884 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135900 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135964 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.135984 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136017 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.136075 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136070 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c32c037-b7a6-4a81-9ce9-478c53e023dd-konnectivity-ca\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136095 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-k8s-cni-cncf-io\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136114 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-bin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136133 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lll85\" (UniqueName: \"kubernetes.io/projected/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-kube-api-access-lll85\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136167 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-kubelet\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136168 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136188 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136243 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136281 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2zzn\" (UniqueName: \"kubernetes.io/projected/62a522bc-31b4-477f-8a93-4d8959c0aabf-kube-api-access-t2zzn\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136320 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136357 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-socket-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136376 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136395 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136449 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136480 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cni-binary-copy\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136510 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-bin\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136538 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mqw\" (UniqueName: \"kubernetes.io/projected/a9b5ab42-4a77-496d-bb0b-b1dd59680622-kube-api-access-g8mqw\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.136554 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136567 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136596 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136639 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/9c32c037-b7a6-4a81-9ce9-478c53e023dd-konnectivity-ca\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136747 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-var-lib-kubelet\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136757 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cni-binary-copy\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136770 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d48d\" (UniqueName: \"kubernetes.io/projected/a577fb95-15eb-4954-91f1-426fca88fb5b-kube-api-access-7d48d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136810 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jbh\" (UniqueName: \"kubernetes.io/projected/02ef4e11-7e6a-49ad-a017-d51735f3b124-kube-api-access-b9jbh\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136877 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136937 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-etc-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136957 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2lxb\" (UniqueName: \"kubernetes.io/projected/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-kube-api-access-r2lxb\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.136984 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137007 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137023 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137038 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-cnibin\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137055 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137079 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137128 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137227 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137241 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-kubelet-config\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137280 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-hostroot\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.137324 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137319 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-host\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137392 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137411 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-binary-copy\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137464 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137467 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137487 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137495 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-netns\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137509 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137540 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-daemon-config\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137667 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-netns\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137697 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-node-log\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137720 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-log-socket\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137756 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-daemon-config\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137764 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-sys\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137805 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137799 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-lib-modules\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137873 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab8f2d0-573d-495b-9075-bced70a5f27b-config-volume\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.137917 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138028 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgnsr\" (UniqueName: \"kubernetes.io/projected/23217ed3-9c56-47de-be14-b34a31cc29e2-kube-api-access-cgnsr\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138050 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138067 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138090 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138116 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02ef4e11-7e6a-49ad-a017-d51735f3b124-hosts-file\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138123 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0ab8f2d0-573d-495b-9075-bced70a5f27b-config-volume\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.138144 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138148 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138172 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-slash\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138250 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138321 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138348 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138369 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138370 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138361 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-script-lib\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138401 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138416 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138418 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138430 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ndvn4\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138424 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138423 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2cqx6\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138464 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d14jkodhae9m6\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138467 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-s9wdq\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138484 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmtf\" (UniqueName: \"kubernetes.io/projected/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-kube-api-access-5nmtf\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138491 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138517 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138522 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138534 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-44jtl\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138557 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138571 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138584 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqhsc\" (UniqueName: \"kubernetes.io/projected/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-kube-api-access-wqhsc\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.138561 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-gmr24\"" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139014 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.139159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139144 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139229 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139342 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-rdfv4\"" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139357 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-script-lib\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139480 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.139848 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgf6s\"" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140036 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140204 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab8f2d0-573d-495b-9075-bced70a5f27b-metrics-tls\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.140303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140256 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.140641 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140451 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 02:56:28.140641 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140478 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.140641 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140540 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.140641 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140580 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140580 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140666 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0ab8f2d0-573d-495b-9075-bced70a5f27b-metrics-tls\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140739 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140742 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140767 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x28m9\"" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140774 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.140852 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140800 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.141161 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140865 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvhbk\" (UniqueName: \"kubernetes.io/projected/5b610fff-d6f5-4856-a97c-04893bee2bb5-kube-api-access-jvhbk\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.141161 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140899 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-tmp\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.141161 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140931 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-serving-certs-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.141161 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.140959 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-config\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.141342 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141253 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 02:56:28.141342 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141309 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-tmp\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.141466 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141420 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.141518 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141491 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.141565 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141533 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.141615 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141577 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.141615 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141586 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 02:56:28.141615 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141589 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf9c99a-3031-4ce6-b564-df6a1feda93c-iptables-alerter-script\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.141748 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141626 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpbxl\" (UniqueName: \"kubernetes.io/projected/9cf9c99a-3031-4ce6-b564-df6a1feda93c-kube-api-access-cpbxl\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.141748 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141655 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-conf-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.141855 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141692 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-ovn\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.141855 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141799 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-metrics-client-ca\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.141959 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141858 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259b75f-532b-4e73-9d4e-0d3d72909feb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141587 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovnkube-config\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.141948 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142109 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysconfig\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142155 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4l8l\" (UniqueName: \"kubernetes.io/projected/4259b75f-532b-4e73-9d4e-0d3d72909feb-kube-api-access-p4l8l\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142192 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142237 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-multus-certs\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142287 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142402 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142475 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-metrics-client-ca\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142499 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-kubernetes\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142653 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142747 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4259b75f-532b-4e73-9d4e-0d3d72909feb-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142788 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-root\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142853 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142908 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142950 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142858 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.142994 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-metrics-client-ca\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143035 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-kubelet\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143059 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9cf9c99a-3031-4ce6-b564-df6a1feda93c-iptables-alerter-script\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143068 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-tuned\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143144 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143199 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143238 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143310 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-tuned\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.143398 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143399 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143441 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143474 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143529 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143555 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143569 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-wtmp\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143618 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzhw\" (UniqueName: \"kubernetes.io/projected/0ab8f2d0-573d-495b-9075-bced70a5f27b-kube-api-access-4pzhw\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143652 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-netd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143676 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143688 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-registration-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143734 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143733 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143767 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143869 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143540 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-metrics-client-ca\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143929 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.143953 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144044 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144093 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144153 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144163 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144374 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144416 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144412 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144528 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144569 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-systemd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144584 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.144640 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144609 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.144723 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-dbus\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145034 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145112 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145159 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5zqc\" (UniqueName: \"kubernetes.io/projected/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kube-api-access-p5zqc\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145285 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145422 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145439 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c32c037-b7a6-4a81-9ce9-478c53e023dd-agent-certs\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:56:28.145811 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145794 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.146196 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145792 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.146196 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.145966 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.146299 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146190 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-tls\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.146299 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146241 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.146299 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146288 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-federate-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.146486 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146456 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.146601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146475 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.146601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146512 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-data-volume\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.146601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146532 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.146601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146586 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.146601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146597 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-data-volume\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.146844 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146676 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab8f2d0-573d-495b-9075-bced70a5f27b-tmp-dir\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.146844 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146704 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/9c32c037-b7a6-4a81-9ce9-478c53e023dd-agent-certs\") pod \"konnectivity-agent-vhplh\" (UID: \"9c32c037-b7a6-4a81-9ce9-478c53e023dd\") " pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:56:28.146844 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146754 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.146844 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146782 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.146844 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146792 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146854 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146860 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146916 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146924 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146976 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146989 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-systemd\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.147118 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.146999 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0ab8f2d0-573d-495b-9075-bced70a5f27b-tmp-dir\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.147430 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147099 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.147430 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147197 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.147430 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147206 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.147430 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147257 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-sys\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.147430 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147354 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jv57\" (UniqueName: \"kubernetes.io/projected/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-kube-api-access-9jv57\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.147430 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147372 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:28.147683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147457 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-socket-dir-parent\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.147683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147504 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-multus\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.147683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147524 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.147683 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.147609 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.148047 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148023 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.148128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148066 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-var-lib-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.148128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148100 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148127 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148131 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148153 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85d227a6-403a-4b90-9750-eb26433e0758-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148163 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148163 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148154 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148211 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.148235 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148231 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148246 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/85d227a6-403a-4b90-9750-eb26433e0758-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148323 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-conf\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148340 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-accelerators-collector-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148356 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-client-tls\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148441 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148503 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148540 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-system-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148546 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5b610fff-d6f5-4856-a97c-04893bee2bb5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148563 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cnibin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.148577 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148572 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148601 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23217ed3-9c56-47de-be14-b34a31cc29e2-serviceca\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148606 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/feaaf2fd-3a0d-46a8-9252-9ea1909150df-original-pull-secret\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148636 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-device-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148676 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148697 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ef4e11-7e6a-49ad-a017-d51735f3b124-tmp-dir\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148691 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148727 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-env-overrides\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148753 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02ef4e11-7e6a-49ad-a017-d51735f3b124-tmp-dir\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148755 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148795 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148861 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148891 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovn-node-metrics-cert\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148919 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-cert\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148938 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148964 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjdmz\" (UniqueName: \"kubernetes.io/projected/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-api-access-vjdmz\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148963 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a9b5ab42-4a77-496d-bb0b-b1dd59680622-env-overrides\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.148987 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.148978 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149009 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149043 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149061 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23217ed3-9c56-47de-be14-b34a31cc29e2-host\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149076 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-sys-fs\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149088 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/62a522bc-31b4-477f-8a93-4d8959c0aabf-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149106 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149127 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149153 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149172 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149357 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23217ed3-9c56-47de-be14-b34a31cc29e2-serviceca\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149375 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149393 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149418 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149432 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149440 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149463 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149653 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-cert\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149654 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149703 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/85d227a6-403a-4b90-9750-eb26433e0758-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149728 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a9b5ab42-4a77-496d-bb0b-b1dd59680622-ovn-node-metrics-cert\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.149798 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.149791 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/4259b75f-532b-4e73-9d4e-0d3d72909feb-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.152119 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.152097 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-serving-certs-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.153082 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.153059 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 02:56:28.153218 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.153197 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 02:56:28.153281 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.153264 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 02:56:28.153645 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.153625 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 02:56:28.153879 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.153856 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.153977 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.153963 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 02:56:28.156880 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.156862 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.156966 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.156925 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.156966 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.156925 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.157161 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.157143 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.157283 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.157265 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.158576 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.158553 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/62a522bc-31b4-477f-8a93-4d8959c0aabf-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.158768 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.158755 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.160393 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.160375 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.160474 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.160405 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.163571 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.163549 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d48d\" (UniqueName: \"kubernetes.io/projected/a577fb95-15eb-4954-91f1-426fca88fb5b-kube-api-access-7d48d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.163668 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.163552 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") pod \"console-5d9f9f777-vk5jc\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:28.164278 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.164258 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.164353 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.164299 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lll85\" (UniqueName: \"kubernetes.io/projected/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-kube-api-access-lll85\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.166138 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.166121 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgnsr\" (UniqueName: \"kubernetes.io/projected/23217ed3-9c56-47de-be14-b34a31cc29e2-kube-api-access-cgnsr\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.167807 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.167789 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2zzn\" (UniqueName: \"kubernetes.io/projected/62a522bc-31b4-477f-8a93-4d8959c0aabf-kube-api-access-t2zzn\") pod \"telemeter-client-5cbbcbd748-fpx9f\" (UID: \"62a522bc-31b4-477f-8a93-4d8959c0aabf\") " pod="openshift-monitoring/telemeter-client-5cbbcbd748-fpx9f" Oct 30 02:56:28.171091 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.171076 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") pod \"alertmanager-main-0\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:56:28.172214 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.172192 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpbxl\" (UniqueName: \"kubernetes.io/projected/9cf9c99a-3031-4ce6-b564-df6a1feda93c-kube-api-access-cpbxl\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.172327 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.172309 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2lxb\" (UniqueName: \"kubernetes.io/projected/c663f5b1-25fa-4b8a-8dc7-95f273fecd53-kube-api-access-r2lxb\") pod \"ingress-canary-hpxr7\" (UID: \"c663f5b1-25fa-4b8a-8dc7-95f273fecd53\") " pod="openshift-ingress-canary/ingress-canary-hpxr7" Oct 30 02:56:28.172372 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.172363 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqhsc\" (UniqueName: \"kubernetes.io/projected/eabb308d-a5a1-4207-bc27-c81e6bb4b2ec-kube-api-access-wqhsc\") pod \"network-metrics-daemon-zmq8r\" (UID: \"eabb308d-a5a1-4207-bc27-c81e6bb4b2ec\") " pod="openshift-multus/network-metrics-daemon-zmq8r" Oct 30 02:56:28.174348 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.174332 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jbh\" (UniqueName: \"kubernetes.io/projected/02ef4e11-7e6a-49ad-a017-d51735f3b124-kube-api-access-b9jbh\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.174529 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.174513 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") pod \"console-7db894548b-j267d\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:28.176220 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.176204 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvhbk\" (UniqueName: \"kubernetes.io/projected/5b610fff-d6f5-4856-a97c-04893bee2bb5-kube-api-access-jvhbk\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.176486 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.176470 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") pod \"prometheus-k8s-0\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:28.177005 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.176989 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjdmz\" (UniqueName: \"kubernetes.io/projected/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-kube-api-access-vjdmz\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.177071 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.177055 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") pod \"console-58955f585-4thl7\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:28.179440 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.179424 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mqw\" (UniqueName: \"kubernetes.io/projected/a9b5ab42-4a77-496d-bb0b-b1dd59680622-kube-api-access-g8mqw\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.180760 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.180743 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4l8l\" (UniqueName: \"kubernetes.io/projected/4259b75f-532b-4e73-9d4e-0d3d72909feb-kube-api-access-p4l8l\") pod \"prometheus-operator-9cd89b74c-lzxzh\" (UID: \"4259b75f-532b-4e73-9d4e-0d3d72909feb\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-lzxzh" Oct 30 02:56:28.180872 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.180846 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkhbh\" (UniqueName: \"kubernetes.io/projected/38027d42-0deb-420c-a777-546f37f24db9-kube-api-access-kkhbh\") pod \"network-check-target-knnkw\" (UID: \"38027d42-0deb-420c-a777-546f37f24db9\") " pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:28.181739 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.181724 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jv57\" (UniqueName: \"kubernetes.io/projected/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-kube-api-access-9jv57\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.182911 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.182897 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzhw\" (UniqueName: \"kubernetes.io/projected/0ab8f2d0-573d-495b-9075-bced70a5f27b-kube-api-access-4pzhw\") pod \"dns-default-gp6sv\" (UID: \"0ab8f2d0-573d-495b-9075-bced70a5f27b\") " pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:28.187171 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.187155 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmtf\" (UniqueName: \"kubernetes.io/projected/eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9-kube-api-access-5nmtf\") pod \"monitoring-plugin-77f4f76594-j69dd\" (UID: \"eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:28.187425 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.187410 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8wsw\" (UniqueName: \"kubernetes.io/projected/85d227a6-403a-4b90-9750-eb26433e0758-kube-api-access-d8wsw\") pod \"kube-state-metrics-5dc8d7d89b-hq4tc\" (UID: \"85d227a6-403a-4b90-9750-eb26433e0758\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-hq4tc" Oct 30 02:56:28.190179 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.190162 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5zqc\" (UniqueName: \"kubernetes.io/projected/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kube-api-access-p5zqc\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249720 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249646 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-device-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249720 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249676 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249720 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249702 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23217ed3-9c56-47de-be14-b34a31cc29e2-host\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.249720 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249721 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-sys-fs\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249738 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-run\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249756 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-crio-socket\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249769 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23217ed3-9c56-47de-be14-b34a31cc29e2-host\") pod \"node-ca-fsp8w\" (UID: \"23217ed3-9c56-47de-be14-b34a31cc29e2\") " pod="openshift-image-registry/node-ca-fsp8w" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249776 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-etc-selinux\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249776 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf9c99a-3031-4ce6-b564-df6a1feda93c-host-slash\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249809 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9cf9c99a-3031-4ce6-b564-df6a1feda93c-host-slash\") pod \"iptables-alerter-7p5r2\" (UID: \"9cf9c99a-3031-4ce6-b564-df6a1feda93c\") " pod="openshift-network-operator/iptables-alerter-7p5r2" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249842 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-run\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249855 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-etc-kubernetes\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249862 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb-crio-socket\") pod \"insights-runtime-extractor-rflzz\" (UID: \"0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb\") " pod="openshift-insights/insights-runtime-extractor-rflzz" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249881 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-systemd-units\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249885 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-etc-kubernetes\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249880 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-sys-fs\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249908 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-modprobe-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249935 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249946 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-device-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249984 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249995 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-modprobe-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.249988 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.249952 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250024 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250034 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-d\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250044 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-k8s-cni-cncf-io\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250062 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-bin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250072 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-systemd-units\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250086 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-kubelet\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250097 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250099 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-k8s-cni-cncf-io\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250107 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-bin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250118 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-socket-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250158 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-bin\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250162 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-kubelet\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250176 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250170 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-socket-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250198 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-bin\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250237 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-var-lib-kubelet\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250249 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250292 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-var-lib-kubelet\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250306 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-etc-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250342 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-etc-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250348 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250376 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-cnibin\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250380 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-system-cni-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250404 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-kubelet-config\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250438 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-kubelet-config\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250441 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-cnibin\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250461 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-hostroot\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250479 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-host\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250493 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-netns\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250508 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-netns\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250532 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-node-log\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250550 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-host\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250555 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-log-socket\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250558 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-netns\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250571 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-hostroot\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250590 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-log-socket\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250593 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-node-log\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.250596 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250583 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-run-netns\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250619 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-sys\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250641 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-lib-modules\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250661 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02ef4e11-7e6a-49ad-a017-d51735f3b124-hosts-file\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250684 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-slash\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250682 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-sys\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250723 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-slash\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250760 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/02ef4e11-7e6a-49ad-a017-d51735f3b124-hosts-file\") pod \"node-resolver-pkc7w\" (UID: \"02ef4e11-7e6a-49ad-a017-d51735f3b124\") " pod="openshift-dns/node-resolver-pkc7w" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250773 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-lib-modules\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250796 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250851 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-conf-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250870 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-ovn\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250889 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysconfig\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250894 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-conf-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250870 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250906 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-multus-certs\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250937 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-run-multus-certs\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250945 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysconfig\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250922 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-ovn\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250963 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-kubernetes\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250992 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-kubernetes\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.250996 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-root\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251017 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251033 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-kubelet\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251037 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-root\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251063 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-wtmp\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251079 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-netd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251094 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-registration-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251099 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251103 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-kubelet\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251120 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-host-cni-netd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251120 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-node-exporter-wtmp\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251131 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-systemd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251168 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-dbus\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251139 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2a5034dc-e3d7-4af9-a808-d2ffad6ba46b-registration-dir\") pod \"aws-ebs-csi-driver-node-m7wsk\" (UID: \"2a5034dc-e3d7-4af9-a808-d2ffad6ba46b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m7wsk" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251196 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251219 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-systemd\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251222 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/feaaf2fd-3a0d-46a8-9252-9ea1909150df-dbus\") pod \"global-pull-secret-syncer-569p6\" (UID: \"feaaf2fd-3a0d-46a8-9252-9ea1909150df\") " pod="kube-system/global-pull-secret-syncer-569p6" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251189 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-run-systemd\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251233 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-sys\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251246 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5b610fff-d6f5-4856-a97c-04893bee2bb5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4t8gf\" (UID: \"5b610fff-d6f5-4856-a97c-04893bee2bb5\") " pod="openshift-multus/multus-additional-cni-plugins-4t8gf" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251256 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d25c69c2-ac9b-492c-9332-bd1e3abd1f1b-sys\") pod \"node-exporter-ffn7c\" (UID: \"d25c69c2-ac9b-492c-9332-bd1e3abd1f1b\") " pod="openshift-monitoring/node-exporter-ffn7c" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251257 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-socket-dir-parent\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251289 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-multus-socket-dir-parent\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251287 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-multus\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251297 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-systemd\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251312 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-var-lib-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251315 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-host-var-lib-cni-multus\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251343 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-conf\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251362 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-system-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251377 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cnibin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251447 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-system-cni-dir\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251488 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a9b5ab42-4a77-496d-bb0b-b1dd59680622-var-lib-openvswitch\") pod \"ovnkube-node-lczmd\" (UID: \"a9b5ab42-4a77-496d-bb0b-b1dd59680622\") " pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251488 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a577fb95-15eb-4954-91f1-426fca88fb5b-etc-sysctl-conf\") pod \"tuned-nlh5q\" (UID: \"a577fb95-15eb-4954-91f1-426fca88fb5b\") " pod="openshift-cluster-node-tuning-operator/tuned-nlh5q" Oct 30 02:56:28.251885 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.251500 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c117c1fb-3ed4-4405-80a7-2fc1613bb98f-cnibin\") pod \"multus-zdmr4\" (UID: \"c117c1fb-3ed4-4405-80a7-2fc1613bb98f\") " pod="openshift-multus/multus-zdmr4" Oct 30 02:56:28.401303 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.401263 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:28.413143 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:28.413117 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gp6sv" Oct 30 02:56:29.959300 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:29.959258 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:29.964278 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:29.964253 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:32.176230 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:32.176193 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:32.191533 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:32.191510 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:56:34.160513 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:34.160471 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:56:36.309834 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:36.309779 5902 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:36.585586 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:36.585513 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-j69dd" Oct 30 02:56:36.995001 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:36.994974 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:56:36.995286 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:36.995230 5902 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 02:56:41.718191 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:56:41.718151 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 02:56:46.314394 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:46.314357 5902 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:56:53.043651 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.043600 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7db894548b-j267d" podUID="68dc07a5-b6fe-4cac-a23b-e753b406161c" containerName="console" containerID="cri-o://58aa88fb7218ec27897780abbe8825f3835fe53c53917d85a9d24a8eef8e6a6d" gracePeriod=15 Oct 30 02:56:53.054216 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.054165 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-58955f585-4thl7" podUID="2a50d657-6d01-4305-b6f5-06eeaa081027" containerName="console" containerID="cri-o://68d8c78fd2c127de2aec31354da628787a22b15e7356616740c6596499eb6e2e" gracePeriod=15 Oct 30 02:56:53.186445 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.186339 5902 generic.go:356] "Generic (PLEG): container finished" podID="2a50d657-6d01-4305-b6f5-06eeaa081027" containerID="68d8c78fd2c127de2aec31354da628787a22b15e7356616740c6596499eb6e2e" exitCode=0 Oct 30 02:56:53.186445 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.186408 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58955f585-4thl7" event={"ID":"2a50d657-6d01-4305-b6f5-06eeaa081027","Type":"ContainerDied","Data":"68d8c78fd2c127de2aec31354da628787a22b15e7356616740c6596499eb6e2e"} Oct 30 02:56:53.188021 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.187999 5902 generic.go:356] "Generic (PLEG): container finished" podID="68dc07a5-b6fe-4cac-a23b-e753b406161c" containerID="58aa88fb7218ec27897780abbe8825f3835fe53c53917d85a9d24a8eef8e6a6d" exitCode=0 Oct 30 02:56:53.188110 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.188077 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db894548b-j267d" event={"ID":"68dc07a5-b6fe-4cac-a23b-e753b406161c","Type":"ContainerDied","Data":"58aa88fb7218ec27897780abbe8825f3835fe53c53917d85a9d24a8eef8e6a6d"} Oct 30 02:56:53.346038 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.346013 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:53.358180 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.358157 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:53.477895 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.477861 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478081 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.477908 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478081 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.477935 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478081 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.477974 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478081 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.477999 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478081 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478028 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478081 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478055 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478306 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478087 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478520 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478494 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478594 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478491 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:56:53.478594 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478555 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478699 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478610 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478699 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478657 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478705 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.478842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478748 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478794 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") pod \"2a50d657-6d01-4305-b6f5-06eeaa081027\" (UID: \"2a50d657-6d01-4305-b6f5-06eeaa081027\") " Oct 30 02:56:53.478998 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478869 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") pod \"68dc07a5-b6fe-4cac-a23b-e753b406161c\" (UID: \"68dc07a5-b6fe-4cac-a23b-e753b406161c\") " Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478693 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478767 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480158 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.478880 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.479127 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.479691 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.479707 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480035 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480436 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480115 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480688 5902 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480717 5902 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a50d657-6d01-4305-b6f5-06eeaa081027-tmp\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480733 5902 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480748 5902 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480762 5902 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480782 5902 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480794 5902 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480808 5902 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480835 5902 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/68dc07a5-b6fe-4cac-a23b-e753b406161c-tmp\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.480995 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.480853 5902 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a50d657-6d01-4305-b6f5-06eeaa081027-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.482411 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.482163 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "kube-api-access-w9khp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:56:53.485031 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.482573 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:53.485031 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.483068 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert" pod "2a50d657-6d01-4305-b6f5-06eeaa081027" (UID: "2a50d657-6d01-4305-b6f5-06eeaa081027"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:53.485031 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.483108 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:53.485031 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.483847 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:56:53.485445 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.485422 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf" pod "68dc07a5-b6fe-4cac-a23b-e753b406161c" (UID: "68dc07a5-b6fe-4cac-a23b-e753b406161c"). InnerVolumeSpecName "kube-api-access-g28pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:56:53.581731 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.581645 5902 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.581731 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.581678 5902 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a50d657-6d01-4305-b6f5-06eeaa081027-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.581731 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.581689 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9khp\" (UniqueName: \"kubernetes.io/projected/2a50d657-6d01-4305-b6f5-06eeaa081027-kube-api-access-w9khp\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.581731 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.581698 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g28pf\" (UniqueName: \"kubernetes.io/projected/68dc07a5-b6fe-4cac-a23b-e753b406161c-kube-api-access-g28pf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.581731 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.581709 5902 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:53.581731 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:53.581718 5902 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/68dc07a5-b6fe-4cac-a23b-e753b406161c-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:56:54.192960 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.192929 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58955f585-4thl7" Oct 30 02:56:54.193364 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.192928 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58955f585-4thl7" event={"ID":"2a50d657-6d01-4305-b6f5-06eeaa081027","Type":"ContainerDied","Data":"f75dbbb006632ffee4dbbaa3bf4eecc9f8173d67418c7f9e5e7751d1a31b741d"} Oct 30 02:56:54.193364 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.193087 5902 scope.go:122] "RemoveContainer" containerID="68d8c78fd2c127de2aec31354da628787a22b15e7356616740c6596499eb6e2e" Oct 30 02:56:54.194438 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.194261 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7db894548b-j267d" event={"ID":"68dc07a5-b6fe-4cac-a23b-e753b406161c","Type":"ContainerDied","Data":"0ae90a8754e1b82562c3ab3d214e89915129952a7b4ef1ae14b868d829600ab2"} Oct 30 02:56:54.194438 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.194307 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7db894548b-j267d" Oct 30 02:56:54.201717 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.201700 5902 scope.go:122] "RemoveContainer" containerID="58aa88fb7218ec27897780abbe8825f3835fe53c53917d85a9d24a8eef8e6a6d" Oct 30 02:56:54.217444 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.217427 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vhplh" Oct 30 02:56:54.240908 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.240887 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58955f585-4thl7"] Oct 30 02:56:54.251799 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.251780 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58955f585-4thl7"] Oct 30 02:56:54.278278 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.278256 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7db894548b-j267d"] Oct 30 02:56:54.283304 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:54.283284 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7db894548b-j267d"] Oct 30 02:56:55.018258 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:55.018224 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a50d657-6d01-4305-b6f5-06eeaa081027" path="/var/lib/kubelet/pods/2a50d657-6d01-4305-b6f5-06eeaa081027/volumes" Oct 30 02:56:55.018672 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:55.018658 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68dc07a5-b6fe-4cac-a23b-e753b406161c" path="/var/lib/kubelet/pods/68dc07a5-b6fe-4cac-a23b-e753b406161c/volumes" Oct 30 02:56:57.017498 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:56:57.017474 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lczmd" Oct 30 02:57:00.282335 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:00.282300 5902 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:01.389452 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.389415 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:57:01.390025 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.389980 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="alertmanager" containerID="cri-o://565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f" gracePeriod=120 Oct 30 02:57:01.390113 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.389991 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy" containerID="cri-o://7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1" gracePeriod=120 Oct 30 02:57:01.390113 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.390080 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-metric" containerID="cri-o://7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f" gracePeriod=120 Oct 30 02:57:01.391037 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.390565 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-web" containerID="cri-o://30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662" gracePeriod=120 Oct 30 02:57:01.391037 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.390906 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="prom-label-proxy" containerID="cri-o://a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16" gracePeriod=120 Oct 30 02:57:01.391152 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:01.391050 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="config-reloader" containerID="cri-o://80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a" gracePeriod=120 Oct 30 02:57:02.228145 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228109 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16" exitCode=0 Oct 30 02:57:02.228145 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228136 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f" exitCode=0 Oct 30 02:57:02.228145 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228142 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1" exitCode=0 Oct 30 02:57:02.228145 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228147 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a" exitCode=0 Oct 30 02:57:02.228145 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228152 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f" exitCode=0 Oct 30 02:57:02.228442 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228179 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16"} Oct 30 02:57:02.228442 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228268 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f"} Oct 30 02:57:02.228442 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228295 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1"} Oct 30 02:57:02.228442 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228319 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a"} Oct 30 02:57:02.228442 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.228342 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f"} Oct 30 02:57:02.652165 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.652142 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:02.764673 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764591 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764673 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764632 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764673 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764660 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764696 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764725 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764750 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764806 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764872 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764928 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.764971 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.764959 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.765321 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.765024 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.765321 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.765081 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.765321 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.765110 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") pod \"d172e77b-1700-49d4-8513-07b3a516b821\" (UID: \"d172e77b-1700-49d4-8513-07b3a516b821\") " Oct 30 02:57:02.765484 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.765348 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:02.765926 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.765898 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:57:02.766381 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.766349 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:02.768405 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.768256 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.768631 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.768603 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "kube-api-access-cchsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:02.769057 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.769031 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.769510 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.769483 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:02.769608 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.769499 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:57:02.769668 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.769625 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.769847 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.769800 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.769984 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.769965 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.774530 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.774504 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.780506 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.780479 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config" pod "d172e77b-1700-49d4-8513-07b3a516b821" (UID: "d172e77b-1700-49d4-8513-07b3a516b821"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:02.866616 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866572 5902 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-cluster-tls-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866616 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866613 5902 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866616 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866624 5902 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-main-db\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866634 5902 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-metrics-client-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866643 5902 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-main-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866653 5902 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-config-volume\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866661 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cchsn\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-kube-api-access-cchsn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866670 5902 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866679 5902 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d172e77b-1700-49d4-8513-07b3a516b821-tls-assets\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866687 5902 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d172e77b-1700-49d4-8513-07b3a516b821-config-out\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866695 5902 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-web-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866703 5902 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d172e77b-1700-49d4-8513-07b3a516b821-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:02.866842 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:02.866711 5902 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d172e77b-1700-49d4-8513-07b3a516b821-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:03.234922 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.234889 5902 generic.go:356] "Generic (PLEG): container finished" podID="d172e77b-1700-49d4-8513-07b3a516b821" containerID="30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662" exitCode=0 Oct 30 02:57:03.235102 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.234942 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662"} Oct 30 02:57:03.235102 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.234964 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.235102 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.235007 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d172e77b-1700-49d4-8513-07b3a516b821","Type":"ContainerDied","Data":"6b5f8544533266e1b48d50fb792edc213add274640579cee9d3906c3def5fbcb"} Oct 30 02:57:03.235102 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.235042 5902 scope.go:122] "RemoveContainer" containerID="a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16" Oct 30 02:57:03.242751 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.242723 5902 scope.go:122] "RemoveContainer" containerID="7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f" Oct 30 02:57:03.249329 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.249314 5902 scope.go:122] "RemoveContainer" containerID="7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1" Oct 30 02:57:03.256164 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.256144 5902 scope.go:122] "RemoveContainer" containerID="30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662" Oct 30 02:57:03.262427 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.262410 5902 scope.go:122] "RemoveContainer" containerID="80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a" Oct 30 02:57:03.268845 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.268802 5902 scope.go:122] "RemoveContainer" containerID="565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f" Oct 30 02:57:03.275521 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.275504 5902 scope.go:122] "RemoveContainer" containerID="62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02" Oct 30 02:57:03.282276 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.282259 5902 scope.go:122] "RemoveContainer" containerID="a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16" Oct 30 02:57:03.282523 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.282504 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16\": container with ID starting with a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16 not found: ID does not exist" containerID="a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16" Oct 30 02:57:03.282566 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.282536 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16"} err="failed to get container status \"a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16\": rpc error: code = NotFound desc = could not find container \"a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16\": container with ID starting with a5ea0679525a87afad714bf8e5e9cedba0d5e845ae4cf6c8ba5e79801aa76e16 not found: ID does not exist" Oct 30 02:57:03.282606 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.282566 5902 scope.go:122] "RemoveContainer" containerID="7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f" Oct 30 02:57:03.282787 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.282771 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f\": container with ID starting with 7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f not found: ID does not exist" containerID="7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f" Oct 30 02:57:03.282848 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.282790 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f"} err="failed to get container status \"7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f\": rpc error: code = NotFound desc = could not find container \"7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f\": container with ID starting with 7430837203da46fc0ac4aa84ff329ace7958d34eedb27b4c66c02b922041286f not found: ID does not exist" Oct 30 02:57:03.282848 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.282802 5902 scope.go:122] "RemoveContainer" containerID="7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1" Oct 30 02:57:03.283027 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.283011 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1\": container with ID starting with 7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1 not found: ID does not exist" containerID="7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1" Oct 30 02:57:03.283065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283032 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1"} err="failed to get container status \"7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1\": rpc error: code = NotFound desc = could not find container \"7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1\": container with ID starting with 7069edcb54d7e4fb3f9120bf4d4904ebb204ddf1b83ed614b38325c55a9c0be1 not found: ID does not exist" Oct 30 02:57:03.283065 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283046 5902 scope.go:122] "RemoveContainer" containerID="30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662" Oct 30 02:57:03.283225 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.283208 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662\": container with ID starting with 30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662 not found: ID does not exist" containerID="30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662" Oct 30 02:57:03.283261 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283228 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662"} err="failed to get container status \"30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662\": rpc error: code = NotFound desc = could not find container \"30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662\": container with ID starting with 30551d99a2d0e9af49ab16e3bda9d2303af21b8870f6d4bef59ada625a1ac662 not found: ID does not exist" Oct 30 02:57:03.283261 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283240 5902 scope.go:122] "RemoveContainer" containerID="80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a" Oct 30 02:57:03.283445 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.283430 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a\": container with ID starting with 80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a not found: ID does not exist" containerID="80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a" Oct 30 02:57:03.283485 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283449 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a"} err="failed to get container status \"80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a\": rpc error: code = NotFound desc = could not find container \"80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a\": container with ID starting with 80cc81926c551f782560f0c3eed3dd24f483969a17c7368bbffe4635c5a4eb4a not found: ID does not exist" Oct 30 02:57:03.283485 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283460 5902 scope.go:122] "RemoveContainer" containerID="565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f" Oct 30 02:57:03.283650 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.283635 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f\": container with ID starting with 565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f not found: ID does not exist" containerID="565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f" Oct 30 02:57:03.283692 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283652 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f"} err="failed to get container status \"565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f\": rpc error: code = NotFound desc = could not find container \"565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f\": container with ID starting with 565636217d8f1072457c61d9d333438ba03d1821204e31f1e9308f7c24db350f not found: ID does not exist" Oct 30 02:57:03.283692 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283665 5902 scope.go:122] "RemoveContainer" containerID="62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02" Oct 30 02:57:03.283870 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:03.283854 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02\": container with ID starting with 62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02 not found: ID does not exist" containerID="62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02" Oct 30 02:57:03.283917 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.283875 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02"} err="failed to get container status \"62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02\": rpc error: code = NotFound desc = could not find container \"62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02\": container with ID starting with 62645ee3b1f8c007190f98f2b6f6d9e8db62d9a15f3d8452c46bd36597fcff02 not found: ID does not exist" Oct 30 02:57:03.297573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.297551 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:57:03.308534 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.308511 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:57:03.341036 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341009 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:57:03.341322 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341306 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="alertmanager" Oct 30 02:57:03.341402 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341325 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="alertmanager" Oct 30 02:57:03.341402 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341352 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy" Oct 30 02:57:03.341402 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341361 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy" Oct 30 02:57:03.341402 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341382 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-metric" Oct 30 02:57:03.341402 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341390 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-metric" Oct 30 02:57:03.341402 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341402 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="prom-label-proxy" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341409 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="prom-label-proxy" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341422 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="init-config-reloader" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341429 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="init-config-reloader" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341444 5902 cpu_manager.go:414] "containerMap: removing container" podUID="68dc07a5-b6fe-4cac-a23b-e753b406161c" containerName="console" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341450 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68dc07a5-b6fe-4cac-a23b-e753b406161c" containerName="console" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341462 5902 cpu_manager.go:414] "containerMap: removing container" podUID="2a50d657-6d01-4305-b6f5-06eeaa081027" containerName="console" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341468 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a50d657-6d01-4305-b6f5-06eeaa081027" containerName="console" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341481 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-web" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341488 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-web" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341499 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="config-reloader" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341506 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="config-reloader" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341569 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="init-config-reloader" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341581 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="68dc07a5-b6fe-4cac-a23b-e753b406161c" containerName="console" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341589 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a50d657-6d01-4305-b6f5-06eeaa081027" containerName="console" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341598 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-web" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341606 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="config-reloader" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341616 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="alertmanager" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341627 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341635 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="kube-rbac-proxy-metric" Oct 30 02:57:03.341700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.341643 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d172e77b-1700-49d4-8513-07b3a516b821" containerName="prom-label-proxy" Oct 30 02:57:03.345666 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.345647 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.365468 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.362464 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 02:57:03.365468 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365160 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 02:57:03.365468 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365239 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 02:57:03.365468 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365308 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 02:57:03.365468 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365412 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2cqx6\"" Oct 30 02:57:03.365743 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365462 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 02:57:03.365743 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365525 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 02:57:03.365743 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365588 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 02:57:03.365743 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.365640 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 02:57:03.371720 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.371703 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 02:57:03.377723 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.377699 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:57:03.470813 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.470780 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnh7j\" (UniqueName: \"kubernetes.io/projected/88ee36a1-9206-4764-8fd9-b6c164fcb996-kube-api-access-cnh7j\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.470977 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.470835 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88ee36a1-9206-4764-8fd9-b6c164fcb996-config-out\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.470977 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.470916 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.470977 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.470932 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ee36a1-9206-4764-8fd9-b6c164fcb996-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.470983 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ee36a1-9206-4764-8fd9-b6c164fcb996-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471075 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-config-volume\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471110 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88ee36a1-9206-4764-8fd9-b6c164fcb996-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471142 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-web-config\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471288 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471253 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88ee36a1-9206-4764-8fd9-b6c164fcb996-tls-assets\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471288 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471282 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471407 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471314 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471407 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471362 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.471407 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.471388 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572158 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572088 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572158 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572128 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572355 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572165 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572355 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572234 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnh7j\" (UniqueName: \"kubernetes.io/projected/88ee36a1-9206-4764-8fd9-b6c164fcb996-kube-api-access-cnh7j\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572355 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572260 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88ee36a1-9206-4764-8fd9-b6c164fcb996-config-out\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572355 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572292 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572355 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572314 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ee36a1-9206-4764-8fd9-b6c164fcb996-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572355 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572335 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ee36a1-9206-4764-8fd9-b6c164fcb996-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572644 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572368 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-config-volume\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572644 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572392 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88ee36a1-9206-4764-8fd9-b6c164fcb996-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572644 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572417 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-web-config\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572644 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572438 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88ee36a1-9206-4764-8fd9-b6c164fcb996-tls-assets\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572644 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572453 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.572644 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572571 5902 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 02:57:03.572951 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.572907 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/88ee36a1-9206-4764-8fd9-b6c164fcb996-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.573291 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.573264 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/88ee36a1-9206-4764-8fd9-b6c164fcb996-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.573862 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.573802 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ee36a1-9206-4764-8fd9-b6c164fcb996-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.575390 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.575349 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.575390 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.575353 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/88ee36a1-9206-4764-8fd9-b6c164fcb996-tls-assets\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.575566 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.575463 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.575748 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.575719 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-web-config\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.575851 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.575787 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-config-volume\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.576030 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.576013 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/88ee36a1-9206-4764-8fd9-b6c164fcb996-config-out\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.576148 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.576131 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.576323 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.576308 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.577466 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.577448 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/88ee36a1-9206-4764-8fd9-b6c164fcb996-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.595103 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.595081 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnh7j\" (UniqueName: \"kubernetes.io/projected/88ee36a1-9206-4764-8fd9-b6c164fcb996-kube-api-access-cnh7j\") pod \"alertmanager-main-0\" (UID: \"88ee36a1-9206-4764-8fd9-b6c164fcb996\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.666545 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.666518 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 02:57:03.994409 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:03.994383 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 02:57:03.996832 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:57:03.996782 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ee36a1_9206_4764_8fd9_b6c164fcb996.slice/crio-5bb51570e403634a2b6cc0a80787554ab25496da440d0e6bba087f003bad6d70 WatchSource:0}: Error finding container 5bb51570e403634a2b6cc0a80787554ab25496da440d0e6bba087f003bad6d70: Status 404 returned error can't find the container with id 5bb51570e403634a2b6cc0a80787554ab25496da440d0e6bba087f003bad6d70 Oct 30 02:57:04.163523 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:04.163490 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-knnkw" Oct 30 02:57:04.242247 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:04.242217 5902 generic.go:356] "Generic (PLEG): container finished" podID="88ee36a1-9206-4764-8fd9-b6c164fcb996" containerID="bc890ffb9a6eea09fbf0a5ed9f47b4251119494af7520f961a0940812526d21c" exitCode=0 Oct 30 02:57:04.242425 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:04.242261 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerDied","Data":"bc890ffb9a6eea09fbf0a5ed9f47b4251119494af7520f961a0940812526d21c"} Oct 30 02:57:04.242425 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:04.242310 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"5bb51570e403634a2b6cc0a80787554ab25496da440d0e6bba087f003bad6d70"} Oct 30 02:57:05.018425 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.018395 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d172e77b-1700-49d4-8513-07b3a516b821" path="/var/lib/kubelet/pods/d172e77b-1700-49d4-8513-07b3a516b821/volumes" Oct 30 02:57:05.253129 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.253095 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"51ae0d4cd9a6743b530d8c9cc515482fc9c666ac421beab558e47d5877f194f4"} Oct 30 02:57:05.253268 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.253190 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"b5013de2420e7b0f92baab3f6fd58b4c34aad121dd7aedfadd29fd5f586eab75"} Oct 30 02:57:05.253268 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.253234 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"b1c61f01cbdf3ef02b8c3c4416055899d2be74376e16752c88e3320de539d2a5"} Oct 30 02:57:05.253268 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.253262 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"7ce9de678adf8b0848adc9455a7df867054d739a36905c6bc14e1c5039ce1212"} Oct 30 02:57:05.253383 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.253284 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"dee11692c0aee3f1c2d47d387f787574699050a0863f7d7eb4087c1f9854ccbd"} Oct 30 02:57:05.253383 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.253307 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"88ee36a1-9206-4764-8fd9-b6c164fcb996","Type":"ContainerStarted","Data":"251856e73cc642dcfeaf9d278018bd3f4ead9b0c98bb5a0ee4fceb54ef7f3afa"} Oct 30 02:57:05.331063 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:05.330963 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.318466024 podStartE2EDuration="2.330945111s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.479087ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:05.319298886 +0000 UTC m=+38.416715445" watchObservedRunningTime="2026-10-30 02:57:05.330945111 +0000 UTC m=+38.428361673" Oct 30 02:57:06.077140 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077104 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:57:06.077549 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077491 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="prometheus" containerID="cri-o://5272a50ba406c489ad991ae1b2ebf6fb377de06426bfc6f3ea5f85c9d7ba2b50" gracePeriod=600 Oct 30 02:57:06.077549 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077505 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy" containerID="cri-o://6f276f039487ef0d651cf4b887834990593214ef53c7630e57d1e3d9272bfddf" gracePeriod=600 Oct 30 02:57:06.077691 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077532 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="thanos-sidecar" containerID="cri-o://f7bf51768c919abc553d06047686fdec9d20117b5ada1c87205d112103c887ed" gracePeriod=600 Oct 30 02:57:06.077691 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077546 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="config-reloader" containerID="cri-o://1aa8707385d532c3f8503d01f2a5a70ab1bfad55f712a2d3373f55473ac98811" gracePeriod=600 Oct 30 02:57:06.077691 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077596 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-web" containerID="cri-o://86daec9c600d1147b4b63e67b1c5fe2fc502a28c556484ad2ba42dbc9bd9bd58" gracePeriod=600 Oct 30 02:57:06.077691 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.077537 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://7769afc9b2ee8d7341bd1b4999df938d305e716bc5bd4b7459339227a6160c90" gracePeriod=600 Oct 30 02:57:06.262092 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262050 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="7769afc9b2ee8d7341bd1b4999df938d305e716bc5bd4b7459339227a6160c90" exitCode=0 Oct 30 02:57:06.262092 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262084 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="6f276f039487ef0d651cf4b887834990593214ef53c7630e57d1e3d9272bfddf" exitCode=0 Oct 30 02:57:06.262092 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262096 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="86daec9c600d1147b4b63e67b1c5fe2fc502a28c556484ad2ba42dbc9bd9bd58" exitCode=0 Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262106 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="f7bf51768c919abc553d06047686fdec9d20117b5ada1c87205d112103c887ed" exitCode=0 Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262115 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="1aa8707385d532c3f8503d01f2a5a70ab1bfad55f712a2d3373f55473ac98811" exitCode=0 Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262127 5902 generic.go:356] "Generic (PLEG): container finished" podID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerID="5272a50ba406c489ad991ae1b2ebf6fb377de06426bfc6f3ea5f85c9d7ba2b50" exitCode=0 Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262142 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"7769afc9b2ee8d7341bd1b4999df938d305e716bc5bd4b7459339227a6160c90"} Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262236 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"6f276f039487ef0d651cf4b887834990593214ef53c7630e57d1e3d9272bfddf"} Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262266 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"86daec9c600d1147b4b63e67b1c5fe2fc502a28c556484ad2ba42dbc9bd9bd58"} Oct 30 02:57:06.262308 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262291 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"f7bf51768c919abc553d06047686fdec9d20117b5ada1c87205d112103c887ed"} Oct 30 02:57:06.262553 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262323 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"1aa8707385d532c3f8503d01f2a5a70ab1bfad55f712a2d3373f55473ac98811"} Oct 30 02:57:06.262553 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.262361 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"5272a50ba406c489ad991ae1b2ebf6fb377de06426bfc6f3ea5f85c9d7ba2b50"} Oct 30 02:57:06.320437 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.320414 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:06.398691 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398618 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.398691 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398656 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.398914 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398692 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.398914 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398805 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.398914 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398867 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.398914 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398903 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398925 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398956 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.398981 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399003 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399040 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399085 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399128 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399112 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399142 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399142 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399175 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399203 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399232 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399193 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399267 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399321 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399460 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399349 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") pod \"d3601216-ec2f-45de-9685-f97ac814e4a7\" (UID: \"d3601216-ec2f-45de-9685-f97ac814e4a7\") " Oct 30 02:57:06.399926 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399504 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.399926 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399798 5902 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.399926 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399834 5902 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.399926 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.399850 5902 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-metrics-client-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.400568 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.400246 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:57:06.401601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.400720 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.401601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.401109 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.401601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.401500 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.401848 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.401602 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.402078 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.401957 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.402167 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.402080 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.402715 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.402694 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.402715 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.402700 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:57:06.402876 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.402856 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.403571 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.403535 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:06.403690 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.403573 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.404122 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.404093 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:57:06.404595 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.404568 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.404678 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.404623 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "kube-api-access-gn2f4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:57:06.404903 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.404873 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.413970 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.413947 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config" pod "d3601216-ec2f-45de-9685-f97ac814e4a7" (UID: "d3601216-ec2f-45de-9685-f97ac814e4a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:57:06.501700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501663 5902 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-metrics-client-certs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501700 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501699 5902 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501712 5902 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-grpc-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501722 5902 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501731 5902 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501740 5902 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501750 5902 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-db\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501760 5902 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d3601216-ec2f-45de-9685-f97ac814e4a7-config-out\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501769 5902 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501777 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn2f4\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-kube-api-access-gn2f4\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501785 5902 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d3601216-ec2f-45de-9685-f97ac814e4a7-tls-assets\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501794 5902 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-kube-rbac-proxy\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501802 5902 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501810 5902 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-web-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501833 5902 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d3601216-ec2f-45de-9685-f97ac814e4a7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501846 5902 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:06.501916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:06.501859 5902 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d3601216-ec2f-45de-9685-f97ac814e4a7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:57:07.269007 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.268975 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d3601216-ec2f-45de-9685-f97ac814e4a7","Type":"ContainerDied","Data":"d3c78350571a9433fc2063c8190087403fb158eb37ba8795dbd952b30661eef8"} Oct 30 02:57:07.269007 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.269003 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.269463 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.269057 5902 scope.go:122] "RemoveContainer" containerID="7769afc9b2ee8d7341bd1b4999df938d305e716bc5bd4b7459339227a6160c90" Oct 30 02:57:07.276654 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.276635 5902 scope.go:122] "RemoveContainer" containerID="6f276f039487ef0d651cf4b887834990593214ef53c7630e57d1e3d9272bfddf" Oct 30 02:57:07.283409 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.283389 5902 scope.go:122] "RemoveContainer" containerID="86daec9c600d1147b4b63e67b1c5fe2fc502a28c556484ad2ba42dbc9bd9bd58" Oct 30 02:57:07.289602 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.289588 5902 scope.go:122] "RemoveContainer" containerID="f7bf51768c919abc553d06047686fdec9d20117b5ada1c87205d112103c887ed" Oct 30 02:57:07.296048 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.296026 5902 scope.go:122] "RemoveContainer" containerID="1aa8707385d532c3f8503d01f2a5a70ab1bfad55f712a2d3373f55473ac98811" Oct 30 02:57:07.302681 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.302655 5902 scope.go:122] "RemoveContainer" containerID="5272a50ba406c489ad991ae1b2ebf6fb377de06426bfc6f3ea5f85c9d7ba2b50" Oct 30 02:57:07.309613 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.309582 5902 scope.go:122] "RemoveContainer" containerID="c130d96630037f1c2d0f3333672ef1308f65196d728dd99ff514a37600d042ef" Oct 30 02:57:07.331177 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.331155 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:57:07.342061 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.342041 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:57:07.368415 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368391 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:57:07.368664 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368651 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="thanos-sidecar" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368665 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="thanos-sidecar" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368675 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="prometheus" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368681 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="prometheus" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368691 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368696 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368704 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="config-reloader" Oct 30 02:57:07.368711 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368708 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="config-reloader" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368719 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-web" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368725 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-web" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368738 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-thanos" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368743 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-thanos" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368750 5902 cpu_manager.go:414] "containerMap: removing container" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="init-config-reloader" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368754 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="init-config-reloader" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368799 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368806 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="config-reloader" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368828 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-web" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368840 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="kube-rbac-proxy-thanos" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368849 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="init-config-reloader" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368858 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="thanos-sidecar" Oct 30 02:57:07.368937 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.368864 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" containerName="prometheus" Oct 30 02:57:07.372752 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.372734 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.387641 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.387616 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 02:57:07.387744 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.387726 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 02:57:07.387947 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.387933 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 02:57:07.387986 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.387944 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 02:57:07.387986 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.387963 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 02:57:07.387986 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.387971 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 02:57:07.388130 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388116 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 02:57:07.388130 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388126 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 02:57:07.388229 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388139 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 02:57:07.388229 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388174 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-d14jkodhae9m6\"" Oct 30 02:57:07.388229 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388225 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 02:57:07.388344 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388248 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-rgf6s\"" Oct 30 02:57:07.388344 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388255 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 02:57:07.388344 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388267 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 02:57:07.388344 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.388327 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 02:57:07.399784 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.399764 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 02:57:07.399784 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.399763 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 02:57:07.403142 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.403104 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:57:07.511784 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.511745 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.511784 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.511788 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-config\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.511972 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.511813 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.511972 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.511922 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512059 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.511980 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512059 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512027 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0beb25d6-9d3f-48e7-a846-36680fa37e88-config-out\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512164 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512059 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512164 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512118 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512164 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512137 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512164 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512153 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512280 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512171 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512280 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512187 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddsvs\" (UniqueName: \"kubernetes.io/projected/0beb25d6-9d3f-48e7-a846-36680fa37e88-kube-api-access-ddsvs\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512280 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512219 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512361 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512303 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512361 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512352 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512421 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512372 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0beb25d6-9d3f-48e7-a846-36680fa37e88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512421 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512396 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512421 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512416 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512502 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512443 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.512502 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.512467 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-web-config\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613103 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613029 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613103 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613066 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613103 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613084 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613338 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613106 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613338 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613130 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddsvs\" (UniqueName: \"kubernetes.io/projected/0beb25d6-9d3f-48e7-a846-36680fa37e88-kube-api-access-ddsvs\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613338 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613155 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613338 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613185 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613338 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613210 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613338 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613240 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0beb25d6-9d3f-48e7-a846-36680fa37e88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613621 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613486 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613621 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613539 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613621 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613569 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613621 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613598 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-web-config\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613636 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613669 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-config\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613712 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613733 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613742 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613764 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.613814 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613794 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0beb25d6-9d3f-48e7-a846-36680fa37e88-config-out\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.614221 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613814 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.614221 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.613886 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.614325 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.614270 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.614507 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.614486 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.615670 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.615645 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.616665 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.616367 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.616665 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.616405 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.616665 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.616564 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.616665 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.616574 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0beb25d6-9d3f-48e7-a846-36680fa37e88-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.616950 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.616888 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.616950 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.616915 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.618099 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.617506 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.618099 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.617672 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.618099 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.618052 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-config\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.618327 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.618304 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-web-config\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.618900 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.618872 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.619248 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.619229 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0beb25d6-9d3f-48e7-a846-36680fa37e88-config-out\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.619627 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.619605 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0beb25d6-9d3f-48e7-a846-36680fa37e88-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.620454 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.620437 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0beb25d6-9d3f-48e7-a846-36680fa37e88-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.636187 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.636167 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddsvs\" (UniqueName: \"kubernetes.io/projected/0beb25d6-9d3f-48e7-a846-36680fa37e88-kube-api-access-ddsvs\") pod \"prometheus-k8s-0\" (UID: \"0beb25d6-9d3f-48e7-a846-36680fa37e88\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:07.693169 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:07.693140 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:08.024038 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:08.023996 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 02:57:08.026078 ip-10-0-142-236 kubenswrapper[5902]: W1030 02:57:08.026052 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0beb25d6_9d3f_48e7_a846_36680fa37e88.slice/crio-9d8825d639c5ea821f8896c01f1effe14f16a5f10c014c434535cce78c352e43 WatchSource:0}: Error finding container 9d8825d639c5ea821f8896c01f1effe14f16a5f10c014c434535cce78c352e43: Status 404 returned error can't find the container with id 9d8825d639c5ea821f8896c01f1effe14f16a5f10c014c434535cce78c352e43 Oct 30 02:57:08.275494 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:08.275407 5902 generic.go:356] "Generic (PLEG): container finished" podID="0beb25d6-9d3f-48e7-a846-36680fa37e88" containerID="2761bc7de55ffad8675fe040260d91fb9ed4a36eddb5a381cd7d131d33732dfd" exitCode=0 Oct 30 02:57:08.275494 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:08.275455 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerDied","Data":"2761bc7de55ffad8675fe040260d91fb9ed4a36eddb5a381cd7d131d33732dfd"} Oct 30 02:57:08.275916 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:08.275530 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"9d8825d639c5ea821f8896c01f1effe14f16a5f10c014c434535cce78c352e43"} Oct 30 02:57:09.018240 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.018207 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3601216-ec2f-45de-9685-f97ac814e4a7" path="/var/lib/kubelet/pods/d3601216-ec2f-45de-9685-f97ac814e4a7/volumes" Oct 30 02:57:09.282228 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.282149 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"fdeb153cabed0424b68f738df4a3eb385f223341407ed6bb943079a29810ff75"} Oct 30 02:57:09.282228 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.282215 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"33def68aa048a7f9ad0f09d5985d1fe0135e8635449a3cfea2274a815dcdeebf"} Oct 30 02:57:09.282599 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.282241 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"85a088513a7a725acfcba0dd28daa52396c58b65eaceb3a661d35ae0727d8833"} Oct 30 02:57:09.282599 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.282265 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"09d53f6e642faab415e015a1f435d18e25b23b28018a24b0875996e7463725d2"} Oct 30 02:57:09.282599 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.282290 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"11cde1c7883813cfab34cd227f1406cc005df7cffed24436fbeee8ece5e739b1"} Oct 30 02:57:09.282599 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.282314 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0beb25d6-9d3f-48e7-a846-36680fa37e88","Type":"ContainerStarted","Data":"af3324d609cd4edeec2db075676ce07e95c5f949514412b092044ed27670b381"} Oct 30 02:57:09.347601 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:09.347554 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.330135774 podStartE2EDuration="2.347541117s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.405343ms" isStatefulPod=false podCreationTimestamp="2026-10-30 02:57:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 02:57:09.346116974 +0000 UTC m=+42.443533533" watchObservedRunningTime="2026-10-30 02:57:09.347541117 +0000 UTC m=+42.444957675" Oct 30 02:57:11.719103 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:11.719066 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 02:57:12.694090 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:12.694053 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:35.692411 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:35.692380 5902 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:35.706903 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:35.706880 5902 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:36.392341 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:36.392316 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 02:57:41.719871 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:57:41.719812 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 02:57:45.260489 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:57:45.260451 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9f9f777-vk5jc"] Oct 30 02:58:10.279638 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.279554 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5d9f9f777-vk5jc" podUID="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" containerName="console" containerID="cri-o://79b81e373dcda5fc834fea15a76e34d376ab42aabb61a406f7c012b10a8f981a" gracePeriod=15 Oct 30 02:58:10.509453 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.509425 5902 generic.go:356] "Generic (PLEG): container finished" podID="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" containerID="79b81e373dcda5fc834fea15a76e34d376ab42aabb61a406f7c012b10a8f981a" exitCode=0 Oct 30 02:58:10.509573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.509495 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9f777-vk5jc" event={"ID":"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14","Type":"ContainerDied","Data":"79b81e373dcda5fc834fea15a76e34d376ab42aabb61a406f7c012b10a8f981a"} Oct 30 02:58:10.509573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.509542 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d9f9f777-vk5jc" event={"ID":"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14","Type":"ContainerDied","Data":"e345b3a0960da3a4969a9f4c307169a770db0fe50b76f505c5b8dd3210ef79d1"} Oct 30 02:58:10.509573 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.509555 5902 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e345b3a0960da3a4969a9f4c307169a770db0fe50b76f505c5b8dd3210ef79d1" Oct 30 02:58:10.515569 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.515548 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:58:10.639740 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639650 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.639740 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639722 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640069 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639754 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640069 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639803 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640069 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639870 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640069 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639905 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640069 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639936 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640069 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.639961 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") pod \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\" (UID: \"c1a1a35c-0d8c-4b3a-a749-17d966ea4d14\") " Oct 30 02:58:10.640346 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.640129 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:10.640346 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.640249 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:10.640423 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.640355 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:10.640511 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.640484 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 02:58:10.640575 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.640543 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 02:58:10.642281 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.642254 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:58:10.642381 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.642308 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 02:58:10.642381 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.642309 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz" pod "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" (UID: "c1a1a35c-0d8c-4b3a-a749-17d966ea4d14"). InnerVolumeSpecName "kube-api-access-t2wnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 02:58:10.740944 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740909 5902 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-trusted-ca-bundle\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.740944 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740937 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2wnz\" (UniqueName: \"kubernetes.io/projected/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-kube-api-access-t2wnz\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.740944 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740949 5902 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-tmp\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.741159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740958 5902 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-service-ca\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.741159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740968 5902 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.741159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740976 5902 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-oauth-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.741159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740984 5902 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-console-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:10.741159 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:10.740992 5902 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14-oauth-serving-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 02:58:11.512868 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:11.512838 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d9f9f777-vk5jc" Oct 30 02:58:11.560599 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:11.560569 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d9f9f777-vk5jc"] Oct 30 02:58:11.571636 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:11.571609 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d9f9f777-vk5jc"] Oct 30 02:58:11.719458 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:58:11.719419 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 02:58:13.018306 ip-10-0-142-236 kubenswrapper[5902]: I1030 02:58:13.018271 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" path="/var/lib/kubelet/pods/c1a1a35c-0d8c-4b3a-a749-17d966ea4d14/volumes" Oct 30 02:58:41.719921 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:58:41.719876 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 02:59:11.719372 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:59:11.719337 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 02:59:41.718930 ip-10-0-142-236 kubenswrapper[5902]: E1030 02:59:41.718853 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:00:11.719888 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:00:11.719847 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:00:41.718308 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:00:41.718266 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:01:11.719023 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:01:11.718940 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:01:41.718738 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:01:41.718702 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:02:11.719077 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:02:11.719036 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:02:27.039570 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:02:27.039542 5902 scope.go:122] "RemoveContainer" containerID="79b81e373dcda5fc834fea15a76e34d376ab42aabb61a406f7c012b10a8f981a" Oct 30 03:02:41.719129 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:02:41.719047 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:03:11.718397 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:03:11.718362 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:03:41.719238 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:03:41.719201 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:04:11.719022 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:04:11.718944 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:04:41.718592 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:04:41.718557 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:05:08.129640 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.129606 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8l9g6"] Oct 30 03:05:08.130107 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.129962 5902 cpu_manager.go:414] "containerMap: removing container" podUID="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" containerName="console" Oct 30 03:05:08.130107 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.129974 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" containerName="console" Oct 30 03:05:08.130107 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.130027 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a1a35c-0d8c-4b3a-a749-17d966ea4d14" containerName="console" Oct 30 03:05:08.133230 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.133210 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.147363 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.147340 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\"" Oct 30 03:05:08.147545 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.147362 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-4fl9d\"" Oct 30 03:05:08.147545 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.147383 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\"" Oct 30 03:05:08.147545 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.147514 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\"" Oct 30 03:05:08.147687 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.147573 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\"" Oct 30 03:05:08.153214 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.153196 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8l9g6"] Oct 30 03:05:08.163358 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.163333 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache\") pod \"s3-init-8l9g6\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.163470 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.163378 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64snf\" (UniqueName: \"kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf\") pod \"s3-init-8l9g6\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.263997 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.263961 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64snf\" (UniqueName: \"kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf\") pod \"s3-init-8l9g6\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.264168 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.264025 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache\") pod \"s3-init-8l9g6\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.264361 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.264346 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache\") pod \"s3-init-8l9g6\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.286875 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.286850 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64snf\" (UniqueName: \"kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf\") pod \"s3-init-8l9g6\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.456231 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.456198 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8l9g6" Oct 30 03:05:08.751780 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.751696 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8l9g6"] Oct 30 03:05:08.755317 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:05:08.755285 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode041a096_569a_448a_9a58_c62853f5df1b.slice/crio-12046af799b838057ce07e62d5c1311efcc8479de8bd1e58e9f720cb8999151d WatchSource:0}: Error finding container 12046af799b838057ce07e62d5c1311efcc8479de8bd1e58e9f720cb8999151d: Status 404 returned error can't find the container with id 12046af799b838057ce07e62d5c1311efcc8479de8bd1e58e9f720cb8999151d Oct 30 03:05:08.757212 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.757196 5902 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:05:08.955259 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:08.955217 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8l9g6" event={"ID":"e041a096-569a-448a-9a58-c62853f5df1b","Type":"ContainerStarted","Data":"12046af799b838057ce07e62d5c1311efcc8479de8bd1e58e9f720cb8999151d"} Oct 30 03:05:11.724302 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:05:11.724261 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:05:12.973439 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:12.973405 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8l9g6" event={"ID":"e041a096-569a-448a-9a58-c62853f5df1b","Type":"ContainerStarted","Data":"4aa528eb984629882f1de4d5d5610f12f2f9a51ae9581a35f218725e7a71d4a4"} Oct 30 03:05:13.978526 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:13.978489 5902 generic.go:356] "Generic (PLEG): container finished" podID="e041a096-569a-448a-9a58-c62853f5df1b" containerID="4aa528eb984629882f1de4d5d5610f12f2f9a51ae9581a35f218725e7a71d4a4" exitCode=0 Oct 30 03:05:13.979081 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:13.978573 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8l9g6" event={"ID":"e041a096-569a-448a-9a58-c62853f5df1b","Type":"ContainerDied","Data":"4aa528eb984629882f1de4d5d5610f12f2f9a51ae9581a35f218725e7a71d4a4"} Oct 30 03:05:19.002353 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:19.002317 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8l9g6" event={"ID":"e041a096-569a-448a-9a58-c62853f5df1b","Type":"ContainerStarted","Data":"f8b7d2369f257f92c6a832804e743ff6f03fa0d4aafa279c3cc6d56bcc77dfb4"} Oct 30 03:05:19.042098 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:05:19.042042 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kserve/s3-init-8l9g6" podStartSLOduration=2.107279691 podStartE2EDuration="11.042027492s" totalImagesPullingTime="7.777004822s" totalInitContainerRuntime="1.157742979s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:05:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:05:19.041424907 +0000 UTC m=+532.138841466" watchObservedRunningTime="2026-10-30 03:05:19.042027492 +0000 UTC m=+532.139444051" Oct 30 03:05:41.718541 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:05:41.718439 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:06:11.718382 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:06:11.718345 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:06:24.229416 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:24.229382 5902 generic.go:356] "Generic (PLEG): container finished" podID="e041a096-569a-448a-9a58-c62853f5df1b" containerID="f8b7d2369f257f92c6a832804e743ff6f03fa0d4aafa279c3cc6d56bcc77dfb4" exitCode=0 Oct 30 03:06:24.229989 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:24.229434 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8l9g6" event={"ID":"e041a096-569a-448a-9a58-c62853f5df1b","Type":"ContainerDied","Data":"f8b7d2369f257f92c6a832804e743ff6f03fa0d4aafa279c3cc6d56bcc77dfb4"} Oct 30 03:06:25.350096 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.350065 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8l9g6" Oct 30 03:06:25.420100 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.420069 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache\") pod \"e041a096-569a-448a-9a58-c62853f5df1b\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " Oct 30 03:06:25.420300 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.420149 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf\" (UniqueName: \"kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf\") pod \"e041a096-569a-448a-9a58-c62853f5df1b\" (UID: \"e041a096-569a-448a-9a58-c62853f5df1b\") " Oct 30 03:06:25.422295 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.422238 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf" pod "e041a096-569a-448a-9a58-c62853f5df1b" (UID: "e041a096-569a-448a-9a58-c62853f5df1b"). InnerVolumeSpecName "kube-api-access-64snf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:06:25.438575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.438547 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache" pod "e041a096-569a-448a-9a58-c62853f5df1b" (UID: "e041a096-569a-448a-9a58-c62853f5df1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:06:25.520768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.520695 5902 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e041a096-569a-448a-9a58-c62853f5df1b-model-cache\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:06:25.520768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:25.520721 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64snf\" (UniqueName: \"kubernetes.io/projected/e041a096-569a-448a-9a58-c62853f5df1b-kube-api-access-64snf\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:06:26.237452 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:26.237424 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8l9g6" Oct 30 03:06:26.237620 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:26.237425 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8l9g6" event={"ID":"e041a096-569a-448a-9a58-c62853f5df1b","Type":"ContainerDied","Data":"12046af799b838057ce07e62d5c1311efcc8479de8bd1e58e9f720cb8999151d"} Oct 30 03:06:26.237620 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:26.237538 5902 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12046af799b838057ce07e62d5c1311efcc8479de8bd1e58e9f720cb8999151d" Oct 30 03:06:41.718712 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:06:41.718674 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:06:45.769641 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.769613 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl"] Oct 30 03:06:45.770039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.769931 5902 cpu_manager.go:414] "containerMap: removing container" podUID="e041a096-569a-448a-9a58-c62853f5df1b" containerName="s3-init" Oct 30 03:06:45.770039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.769943 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e041a096-569a-448a-9a58-c62853f5df1b" containerName="s3-init" Oct 30 03:06:45.770039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.769973 5902 cpu_manager.go:414] "containerMap: removing container" podUID="e041a096-569a-448a-9a58-c62853f5df1b" containerName="download-hf-model" Oct 30 03:06:45.770039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.769978 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e041a096-569a-448a-9a58-c62853f5df1b" containerName="download-hf-model" Oct 30 03:06:45.770039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.770026 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="e041a096-569a-448a-9a58-c62853f5df1b" containerName="s3-init" Oct 30 03:06:45.773322 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.773302 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.797642 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797617 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:06:45.797779 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797663 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-rollback-v2-predictor-serving-cert\"" Oct 30 03:06:45.797779 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797663 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"seaweedfs-s3-creds\"" Oct 30 03:06:45.797779 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797663 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\"" Oct 30 03:06:45.797779 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797749 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"openshift-service-ca.crt\"" Oct 30 03:06:45.797779 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797758 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"kube-root-ca.crt\"" Oct 30 03:06:45.797989 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.797806 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"default-dockercfg-57rzd\"" Oct 30 03:06:45.800227 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.800208 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl"] Oct 30 03:06:45.894347 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.894312 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbqg\" (UniqueName: \"kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.894517 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.894362 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.894517 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.894486 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.894588 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.894542 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.894626 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.894590 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.949042 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.949012 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg"] Oct 30 03:06:45.952565 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.952546 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:45.975500 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.975476 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg"] Oct 30 03:06:45.995784 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.995756 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.995936 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.995790 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbqg\" (UniqueName: \"kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.995936 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.995834 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.995936 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.995868 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.995936 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.995910 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.996371 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.996344 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.996538 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.996519 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.996617 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.996599 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:45.998327 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:45.998308 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:46.018894 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.018866 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbqg\" (UniqueName: \"kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg\") pod \"isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:46.019548 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.019522 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-rollback-predictor-serving-cert\"" Oct 30 03:06:46.090298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.090220 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:06:46.096613 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.096586 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.096724 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.096631 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.096785 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.096719 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.096852 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.096785 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjg8w\" (UniqueName: \"kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.096897 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.096880 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198224 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198193 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198386 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198249 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198386 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198273 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198386 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198298 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198386 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198318 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjg8w\" (UniqueName: \"kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198735 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198711 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.198989 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.198964 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.199098 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.199019 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.200662 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.200642 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.223299 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.223268 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjg8w\" (UniqueName: \"kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-qq5xg\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.268929 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.268899 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:06:46.401495 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.401473 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl"] Oct 30 03:06:46.404050 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:06:46.404025 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf788ffdc_1b61_438f_9812_41217114b262.slice/crio-87c85254e47945ffc8a3309415a31a301070ef2265b64e152ee81ae80604eda7 WatchSource:0}: Error finding container 87c85254e47945ffc8a3309415a31a301070ef2265b64e152ee81ae80604eda7: Status 404 returned error can't find the container with id 87c85254e47945ffc8a3309415a31a301070ef2265b64e152ee81ae80604eda7 Oct 30 03:06:46.419803 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.419775 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl"] Oct 30 03:06:46.424941 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.424920 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.441265 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.441244 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-raw-sklearn-batcher-cac05-kube-rbac-proxy-sar-config\"" Oct 30 03:06:46.441265 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.441254 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:06:46.441391 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.441263 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"default-dockercfg-f9xch\"" Oct 30 03:06:46.444793 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.444777 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"seaweedfs-s3-creds\"" Oct 30 03:06:46.460764 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.460739 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl"] Oct 30 03:06:46.500885 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.500856 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.501028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.500897 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqk9k\" (UniqueName: \"kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.501028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.500976 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.501100 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.501028 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.501100 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.501079 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.541309 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.541268 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"openshift-service-ca.crt\"" Oct 30 03:06:46.541478 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.541393 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"kube-root-ca.crt\"" Oct 30 03:06:46.541534 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.541522 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-raw-sklearn-batcher-cac05-predictor-serving-cert\"" Oct 30 03:06:46.567474 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.567444 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg"] Oct 30 03:06:46.570722 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:06:46.570693 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f643df2_d227_4fbd_8140_7c2a47ca7e25.slice/crio-4d024b739e6d82d67905916f7af48c26e8811bde08dcf5107f80e0b4a386d06a WatchSource:0}: Error finding container 4d024b739e6d82d67905916f7af48c26e8811bde08dcf5107f80e0b4a386d06a: Status 404 returned error can't find the container with id 4d024b739e6d82d67905916f7af48c26e8811bde08dcf5107f80e0b4a386d06a Oct 30 03:06:46.602054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602020 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602196 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602080 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602196 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602157 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602196 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602189 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602409 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602233 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqk9k\" (UniqueName: \"kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602683 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602657 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602810 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602787 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.602926 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.602904 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.604613 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.604595 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.628698 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.628668 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqk9k\" (UniqueName: \"kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k\") pod \"isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:46.749121 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:46.749091 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:06:47.074896 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.074855 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl"] Oct 30 03:06:47.077097 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:06:47.077067 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddadd5ba7_86a7_45b0_981a_9ff8769e910c.slice/crio-3a60260f8ece7ea2dec3ac6a744e82f13132599f6ab8e4bc7403b1087fa1e5e7 WatchSource:0}: Error finding container 3a60260f8ece7ea2dec3ac6a744e82f13132599f6ab8e4bc7403b1087fa1e5e7: Status 404 returned error can't find the container with id 3a60260f8ece7ea2dec3ac6a744e82f13132599f6ab8e4bc7403b1087fa1e5e7 Oct 30 03:06:47.311579 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.311537 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerStarted","Data":"f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd"} Oct 30 03:06:47.311901 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.311868 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerStarted","Data":"3a60260f8ece7ea2dec3ac6a744e82f13132599f6ab8e4bc7403b1087fa1e5e7"} Oct 30 03:06:47.313270 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.313213 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerStarted","Data":"9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51"} Oct 30 03:06:47.313420 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.313302 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerStarted","Data":"4d024b739e6d82d67905916f7af48c26e8811bde08dcf5107f80e0b4a386d06a"} Oct 30 03:06:47.315074 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.315046 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerStarted","Data":"33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63"} Oct 30 03:06:47.315188 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:47.315106 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerStarted","Data":"87c85254e47945ffc8a3309415a31a301070ef2265b64e152ee81ae80604eda7"} Oct 30 03:06:51.330170 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:51.330090 5902 generic.go:356] "Generic (PLEG): container finished" podID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerID="9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51" exitCode=0 Oct 30 03:06:51.330640 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:51.330163 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerDied","Data":"9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51"} Oct 30 03:06:51.331850 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:51.331807 5902 generic.go:356] "Generic (PLEG): container finished" podID="f788ffdc-1b61-438f-9812-41217114b262" containerID="33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63" exitCode=0 Oct 30 03:06:51.331850 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:51.331846 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerDied","Data":"33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63"} Oct 30 03:06:51.333678 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:51.333656 5902 generic.go:356] "Generic (PLEG): container finished" podID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerID="f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd" exitCode=0 Oct 30 03:06:51.333779 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:06:51.333690 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerDied","Data":"f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd"} Oct 30 03:07:11.723779 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:07:11.722813 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:07:18.594358 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:18.594262 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerStarted","Data":"e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7"} Oct 30 03:07:18.596810 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:18.596778 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerStarted","Data":"e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55"} Oct 30 03:07:18.599073 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:18.599048 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerStarted","Data":"665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a"} Oct 30 03:07:21.616088 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.616003 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerStarted","Data":"87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac"} Oct 30 03:07:21.616476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.616190 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:07:21.616476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.616353 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:07:21.617715 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.617681 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:07:21.618325 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.618305 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerStarted","Data":"8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a"} Oct 30 03:07:21.620172 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.620152 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerStarted","Data":"747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5"} Oct 30 03:07:21.620274 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.620259 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:07:21.670203 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.670154 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podStartSLOduration=3.244100375 podStartE2EDuration="36.670142568s" totalImagesPullingTime="29.564457386s" totalInitContainerRuntime="3.861584807s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:21.669272277 +0000 UTC m=+654.766688837" watchObservedRunningTime="2026-10-30 03:07:21.670142568 +0000 UTC m=+654.767559120" Oct 30 03:07:21.711749 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:21.711698 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podStartSLOduration=3.284968336 podStartE2EDuration="36.711685993s" totalImagesPullingTime="29.544352183s" totalInitContainerRuntime="3.882365474s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:21.710801619 +0000 UTC m=+654.808218202" watchObservedRunningTime="2026-10-30 03:07:21.711685993 +0000 UTC m=+654.809102549" Oct 30 03:07:22.624332 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:22.624289 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:07:22.624717 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:22.624467 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:07:22.625737 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:22.625709 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:07:23.633476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.633441 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerStarted","Data":"9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b"} Oct 30 03:07:23.633992 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.633910 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:07:23.633992 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.633964 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:07:23.634109 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.634018 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:07:23.634109 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.634054 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:07:23.635540 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.635501 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="Get \"http://10.134.0.23:9081/\": dial tcp 10.134.0.23:9081: connect: connection refused" Oct 30 03:07:23.635677 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.635501 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:07:23.698664 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:23.698619 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podStartSLOduration=1.753031328 podStartE2EDuration="37.698607323s" totalImagesPullingTime="32.062287477s" totalInitContainerRuntime="3.883288518s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:06:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:07:23.687763888 +0000 UTC m=+656.785180481" watchObservedRunningTime="2026-10-30 03:07:23.698607323 +0000 UTC m=+656.796023881" Oct 30 03:07:24.636877 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:24.636836 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:07:24.637857 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:24.637805 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:07:25.640130 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:25.640084 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:07:25.640522 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:25.640503 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:07:27.630677 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:27.630649 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:07:27.631339 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:27.631295 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:07:28.638257 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:28.638230 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:07:28.638724 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:28.638701 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:07:30.644702 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:30.644671 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:07:30.645285 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:30.645249 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:07:30.645633 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:30.645611 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:07:37.631696 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:37.631642 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:07:38.639219 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:38.639182 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:07:40.645245 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:40.645202 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:07:40.645689 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:40.645645 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:07:41.719995 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:07:41.719958 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:07:47.631169 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:47.631125 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:07:48.639223 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:48.639183 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:07:50.645598 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:50.645556 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:07:50.646142 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:50.646113 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:07:57.631533 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:57.631483 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:07:58.639433 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:07:58.639395 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:08:00.645268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:00.645229 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:08:00.645720 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:00.645636 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:08:07.631610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:07.631557 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:08:08.638772 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:08.638733 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:08:10.645186 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:10.645142 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:08:10.645668 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:10.645642 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:08:11.719671 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:11.719635 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:08:17.631860 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:17.631781 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:08:18.639615 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:18.639578 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:08:20.645164 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:20.645117 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:08:20.645641 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:20.645515 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:08:27.631135 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:27.631089 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:08:28.639323 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:28.639282 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:08:30.014243 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:30.014197 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:08:30.645921 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:30.645880 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:08:30.646350 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:30.646326 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:08:37.631607 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:37.631502 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused" Oct 30 03:08:40.014853 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:40.014803 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:08:40.646043 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:40.646010 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:08:40.646261 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:40.646246 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:08:41.719655 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:41.719621 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:08:47.632525 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:47.632490 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:08:48.094615 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:48.094582 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg"] Oct 30 03:08:48.094886 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:48.094857 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" containerID="cri-o://665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a" gracePeriod=30 Oct 30 03:08:48.095029 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:48.094865 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kube-rbac-proxy" containerID="cri-o://747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5" gracePeriod=30 Oct 30 03:08:48.633333 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:48.633286 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused" Oct 30 03:08:48.964918 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:48.964882 5902 generic.go:356] "Generic (PLEG): container finished" podID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerID="747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5" exitCode=0 Oct 30 03:08:48.965090 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:48.964923 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerDied","Data":"747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5"} Oct 30 03:08:50.014234 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:50.014196 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused" Oct 30 03:08:50.870268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:50.870233 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl"] Oct 30 03:08:50.870554 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:50.870524 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" containerID="cri-o://e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7" gracePeriod=30 Oct 30 03:08:50.870673 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:50.870558 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kube-rbac-proxy" containerID="cri-o://87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac" gracePeriod=30 Oct 30 03:08:51.091181 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.091145 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8"] Oct 30 03:08:51.094918 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.094889 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.117835 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.117778 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8"] Oct 30 03:08:51.216983 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.216925 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.216983 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.216992 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.217209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.217132 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqzxh\" (UniqueName: \"kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.217209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.217174 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.217209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.217203 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.317851 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.317802 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.317862 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.317907 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqzxh\" (UniqueName: \"kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.317924 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.317960 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318237 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.318216 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318541 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.318523 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.318632 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.318613 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.320484 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.320464 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.341067 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.341036 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqzxh\" (UniqueName: \"kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh\") pod \"isvc-canary-rollback-predictor-598dbf8bc4-mhxr8\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.411982 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.411945 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:51.613979 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.613894 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl"] Oct 30 03:08:51.614159 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.614134 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" containerID="cri-o://e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55" gracePeriod=30 Oct 30 03:08:51.614232 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.614159 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" containerID="cri-o://8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a" gracePeriod=30 Oct 30 03:08:51.614287 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.614140 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" containerID="cri-o://9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b" gracePeriod=30 Oct 30 03:08:51.723800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.723756 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8"] Oct 30 03:08:51.727645 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:08:51.727547 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c6d688_d7e7_4839_a790_0de78007fd46.slice/crio-656b9f88bd494b2a0448615c491d8f62ca9bb121c795e62e0a24fe1f4335b17f WatchSource:0}: Error finding container 656b9f88bd494b2a0448615c491d8f62ca9bb121c795e62e0a24fe1f4335b17f: Status 404 returned error can't find the container with id 656b9f88bd494b2a0448615c491d8f62ca9bb121c795e62e0a24fe1f4335b17f Oct 30 03:08:51.978624 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.978594 5902 generic.go:356] "Generic (PLEG): container finished" podID="f788ffdc-1b61-438f-9812-41217114b262" containerID="87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac" exitCode=0 Oct 30 03:08:51.978797 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.978685 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerDied","Data":"87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac"} Oct 30 03:08:51.980378 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.980343 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerStarted","Data":"a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25"} Oct 30 03:08:51.980551 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.980418 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerStarted","Data":"656b9f88bd494b2a0448615c491d8f62ca9bb121c795e62e0a24fe1f4335b17f"} Oct 30 03:08:51.982887 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.982866 5902 generic.go:356] "Generic (PLEG): container finished" podID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerID="8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a" exitCode=0 Oct 30 03:08:51.982993 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:51.982925 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerDied","Data":"8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a"} Oct 30 03:08:52.624813 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:52.624773 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused" Oct 30 03:08:53.438884 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.438860 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:08:53.538519 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538439 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config\") pod \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " Oct 30 03:08:53.538519 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538481 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls\") pod \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " Oct 30 03:08:53.538744 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538535 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert\") pod \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " Oct 30 03:08:53.538744 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538646 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location\") pod \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " Oct 30 03:08:53.538744 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538675 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w\" (UniqueName: \"kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w\") pod \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\" (UID: \"8f643df2-d227-4fbd-8140-7c2a47ca7e25\") " Oct 30 03:08:53.538933 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538892 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config" pod "8f643df2-d227-4fbd-8140-7c2a47ca7e25" (UID: "8f643df2-d227-4fbd-8140-7c2a47ca7e25"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:53.538991 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538955 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:53.539051 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538986 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert" pod "8f643df2-d227-4fbd-8140-7c2a47ca7e25" (UID: "8f643df2-d227-4fbd-8140-7c2a47ca7e25"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:53.539051 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.538993 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location" pod "8f643df2-d227-4fbd-8140-7c2a47ca7e25" (UID: "8f643df2-d227-4fbd-8140-7c2a47ca7e25"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:08:53.540831 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.540798 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls" pod "8f643df2-d227-4fbd-8140-7c2a47ca7e25" (UID: "8f643df2-d227-4fbd-8140-7c2a47ca7e25"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:53.540931 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.540860 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w" pod "8f643df2-d227-4fbd-8140-7c2a47ca7e25" (UID: "8f643df2-d227-4fbd-8140-7c2a47ca7e25"). InnerVolumeSpecName "kube-api-access-sjg8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:53.639365 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.639331 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f643df2-d227-4fbd-8140-7c2a47ca7e25-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:53.639365 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.639362 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:53.639365 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.639371 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjg8w\" (UniqueName: \"kubernetes.io/projected/8f643df2-d227-4fbd-8140-7c2a47ca7e25-kube-api-access-sjg8w\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:53.639850 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.639381 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f643df2-d227-4fbd-8140-7c2a47ca7e25-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:53.997635 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.997589 5902 generic.go:356] "Generic (PLEG): container finished" podID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerID="665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a" exitCode=0 Oct 30 03:08:53.997799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.997629 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerDied","Data":"665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a"} Oct 30 03:08:53.997799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.997667 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" Oct 30 03:08:53.997799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.997723 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg" event={"ID":"8f643df2-d227-4fbd-8140-7c2a47ca7e25","Type":"ContainerDied","Data":"4d024b739e6d82d67905916f7af48c26e8811bde08dcf5107f80e0b4a386d06a"} Oct 30 03:08:53.997799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:53.997750 5902 scope.go:122] "RemoveContainer" containerID="747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5" Oct 30 03:08:54.005806 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.005788 5902 scope.go:122] "RemoveContainer" containerID="665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a" Oct 30 03:08:54.013345 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.013325 5902 scope.go:122] "RemoveContainer" containerID="9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51" Oct 30 03:08:54.020193 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.020178 5902 scope.go:122] "RemoveContainer" containerID="747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5" Oct 30 03:08:54.020451 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:54.020424 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5\": container with ID starting with 747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5 not found: ID does not exist" containerID="747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5" Oct 30 03:08:54.020510 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.020460 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5"} err="failed to get container status \"747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5\": rpc error: code = NotFound desc = could not find container \"747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5\": container with ID starting with 747fc4e2ca7765d74a1dc050dfb95cdee1cba24eda06f992a85c1318a35d8ce5 not found: ID does not exist" Oct 30 03:08:54.020510 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.020489 5902 scope.go:122] "RemoveContainer" containerID="665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a" Oct 30 03:08:54.020686 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:54.020669 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a\": container with ID starting with 665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a not found: ID does not exist" containerID="665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a" Oct 30 03:08:54.020727 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.020689 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a"} err="failed to get container status \"665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a\": rpc error: code = NotFound desc = could not find container \"665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a\": container with ID starting with 665fc7703cded45825ccc470fbdb24aad8c672e72f25b863dee6c5354d3dcb1a not found: ID does not exist" Oct 30 03:08:54.020727 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.020702 5902 scope.go:122] "RemoveContainer" containerID="9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51" Oct 30 03:08:54.020948 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:54.020927 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51\": container with ID starting with 9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51 not found: ID does not exist" containerID="9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51" Oct 30 03:08:54.020995 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.020955 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51"} err="failed to get container status \"9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51\": rpc error: code = NotFound desc = could not find container \"9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51\": container with ID starting with 9332dd19bc32272652bfb465b36ad2ef043f9541041c5b532eaa2c8d76429b51 not found: ID does not exist" Oct 30 03:08:54.056927 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.056899 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg"] Oct 30 03:08:54.067201 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:54.067175 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-qq5xg"] Oct 30 03:08:55.017623 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:55.017584 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" path="/var/lib/kubelet/pods/8f643df2-d227-4fbd-8140-7c2a47ca7e25/volumes" Oct 30 03:08:55.641216 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:55.641175 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 30 03:08:56.008214 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.008182 5902 generic.go:356] "Generic (PLEG): container finished" podID="57c6d688-d7e7-4839-a790-0de78007fd46" containerID="a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25" exitCode=0 Oct 30 03:08:56.008403 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.008261 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerDied","Data":"a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25"} Oct 30 03:08:56.415389 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.415368 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:08:56.566517 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566440 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location\") pod \"f788ffdc-1b61-438f-9812-41217114b262\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " Oct 30 03:08:56.566517 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566497 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert\") pod \"f788ffdc-1b61-438f-9812-41217114b262\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " Oct 30 03:08:56.566757 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566541 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls\") pod \"f788ffdc-1b61-438f-9812-41217114b262\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " Oct 30 03:08:56.566757 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566609 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg\" (UniqueName: \"kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg\") pod \"f788ffdc-1b61-438f-9812-41217114b262\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " Oct 30 03:08:56.566885 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566785 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config\") pod \"f788ffdc-1b61-438f-9812-41217114b262\" (UID: \"f788ffdc-1b61-438f-9812-41217114b262\") " Oct 30 03:08:56.566885 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566853 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location" pod "f788ffdc-1b61-438f-9812-41217114b262" (UID: "f788ffdc-1b61-438f-9812-41217114b262"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:08:56.566988 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.566961 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert" pod "f788ffdc-1b61-438f-9812-41217114b262" (UID: "f788ffdc-1b61-438f-9812-41217114b262"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:56.567168 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.567146 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config" pod "f788ffdc-1b61-438f-9812-41217114b262" (UID: "f788ffdc-1b61-438f-9812-41217114b262"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:08:56.567268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.567153 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f788ffdc-1b61-438f-9812-41217114b262-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:56.567268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.567187 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:56.568634 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.568612 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg" pod "f788ffdc-1b61-438f-9812-41217114b262" (UID: "f788ffdc-1b61-438f-9812-41217114b262"). InnerVolumeSpecName "kube-api-access-5fbqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:08:56.568735 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.568665 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls" pod "f788ffdc-1b61-438f-9812-41217114b262" (UID: "f788ffdc-1b61-438f-9812-41217114b262"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:08:56.667626 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.667589 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f788ffdc-1b61-438f-9812-41217114b262-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:56.667626 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.667620 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fbqg\" (UniqueName: \"kubernetes.io/projected/f788ffdc-1b61-438f-9812-41217114b262-kube-api-access-5fbqg\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:56.667626 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:56.667630 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f788ffdc-1b61-438f-9812-41217114b262-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:08:57.015559 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.015528 5902 generic.go:356] "Generic (PLEG): container finished" podID="f788ffdc-1b61-438f-9812-41217114b262" containerID="e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7" exitCode=0 Oct 30 03:08:57.015836 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.015798 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" Oct 30 03:08:57.019405 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019381 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerDied","Data":"e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7"} Oct 30 03:08:57.019574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019426 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl" event={"ID":"f788ffdc-1b61-438f-9812-41217114b262","Type":"ContainerDied","Data":"87c85254e47945ffc8a3309415a31a301070ef2265b64e152ee81ae80604eda7"} Oct 30 03:08:57.019574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019455 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerStarted","Data":"f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116"} Oct 30 03:08:57.019574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019478 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerStarted","Data":"f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b"} Oct 30 03:08:57.019574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019509 5902 scope.go:122] "RemoveContainer" containerID="87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac" Oct 30 03:08:57.019792 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019774 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:57.019894 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.019869 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:08:57.024463 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.024121 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:08:57.033557 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.033536 5902 scope.go:122] "RemoveContainer" containerID="e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7" Oct 30 03:08:57.040848 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.040806 5902 scope.go:122] "RemoveContainer" containerID="33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63" Oct 30 03:08:57.050194 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.050171 5902 scope.go:122] "RemoveContainer" containerID="87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac" Oct 30 03:08:57.050809 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:57.050782 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac\": container with ID starting with 87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac not found: ID does not exist" containerID="87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac" Oct 30 03:08:57.050911 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.050837 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac"} err="failed to get container status \"87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac\": rpc error: code = NotFound desc = could not find container \"87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac\": container with ID starting with 87272cd9220c3cb01a0936d5590967f788c0a56de7aec0554d16671d8d1063ac not found: ID does not exist" Oct 30 03:08:57.050911 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.050862 5902 scope.go:122] "RemoveContainer" containerID="e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7" Oct 30 03:08:57.051120 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:57.051103 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7\": container with ID starting with e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7 not found: ID does not exist" containerID="e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7" Oct 30 03:08:57.051176 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.051127 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7"} err="failed to get container status \"e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7\": rpc error: code = NotFound desc = could not find container \"e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7\": container with ID starting with e0910dab7ea69b016289cecb92fe26bc524e1389a147525038b86f23f517aae7 not found: ID does not exist" Oct 30 03:08:57.051176 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.051147 5902 scope.go:122] "RemoveContainer" containerID="33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63" Oct 30 03:08:57.051391 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:08:57.051375 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63\": container with ID starting with 33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63 not found: ID does not exist" containerID="33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63" Oct 30 03:08:57.051437 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.051395 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63"} err="failed to get container status \"33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63\": rpc error: code = NotFound desc = could not find container \"33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63\": container with ID starting with 33250a1b9306f4bdecdb47c5147ad7c3e42e4bdd101278ff4b2adb33524c8d63 not found: ID does not exist" Oct 30 03:08:57.073537 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.073496 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podStartSLOduration=2.307710559 podStartE2EDuration="6.073482877s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.765772318s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:08:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:08:57.073452996 +0000 UTC m=+750.170869556" watchObservedRunningTime="2026-10-30 03:08:57.073482877 +0000 UTC m=+750.170899433" Oct 30 03:08:57.107277 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.107234 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl"] Oct 30 03:08:57.112122 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:57.112095 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-v2-predictor-9d9fddf5c-xjknl"] Oct 30 03:08:58.023952 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:58.023915 5902 generic.go:356] "Generic (PLEG): container finished" podID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerID="e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55" exitCode=0 Oct 30 03:08:58.024362 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:58.023986 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerDied","Data":"e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55"} Oct 30 03:08:58.025295 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:58.025267 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:08:59.018409 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:08:59.018379 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f788ffdc-1b61-438f-9812-41217114b262" path="/var/lib/kubelet/pods/f788ffdc-1b61-438f-9812-41217114b262/volumes" Oct 30 03:09:00.640565 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:00.640526 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 30 03:09:00.645944 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:00.645912 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:09:00.646312 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:00.646287 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:09:03.029482 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:03.029451 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:09:03.030036 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:03.030003 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:09:05.640768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:05.640727 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 30 03:09:05.641268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:05.640871 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:09:10.641079 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:10.641039 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 30 03:09:10.645347 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:10.645314 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:09:10.645713 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:10.645680 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:09:11.719229 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:09:11.719195 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:09:13.029957 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:13.029919 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:09:15.640685 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:15.640643 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 30 03:09:20.640794 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:20.640756 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused" Oct 30 03:09:20.645269 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:20.645238 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused" Oct 30 03:09:20.645401 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:20.645361 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:09:20.645401 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:20.645388 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:09:20.645512 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:20.645499 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:09:21.769039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.769012 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:09:21.880642 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.880562 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config\") pod \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " Oct 30 03:09:21.880642 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.880604 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert\") pod \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " Oct 30 03:09:21.880642 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.880629 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k\" (UniqueName: \"kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k\") pod \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " Oct 30 03:09:21.880902 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.880656 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls\" (UniqueName: \"kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls\") pod \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " Oct 30 03:09:21.880902 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.880703 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location\") pod \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\" (UID: \"dadd5ba7-86a7-45b0-981a-9ff8769e910c\") " Oct 30 03:09:21.881092 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.881067 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config" pod "dadd5ba7-86a7-45b0-981a-9ff8769e910c" (UID: "dadd5ba7-86a7-45b0-981a-9ff8769e910c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:21.881092 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.881077 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location" pod "dadd5ba7-86a7-45b0-981a-9ff8769e910c" (UID: "dadd5ba7-86a7-45b0-981a-9ff8769e910c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:09:21.881186 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.881126 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert" pod "dadd5ba7-86a7-45b0-981a-9ff8769e910c" (UID: "dadd5ba7-86a7-45b0-981a-9ff8769e910c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:09:21.883008 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.882980 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls" pod "dadd5ba7-86a7-45b0-981a-9ff8769e910c" (UID: "dadd5ba7-86a7-45b0-981a-9ff8769e910c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:09:21.883008 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.882993 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k" pod "dadd5ba7-86a7-45b0-981a-9ff8769e910c" (UID: "dadd5ba7-86a7-45b0-981a-9ff8769e910c"). InnerVolumeSpecName "kube-api-access-fqk9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:09:21.982031 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.981995 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:09:21.982031 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.982025 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dadd5ba7-86a7-45b0-981a-9ff8769e910c-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:09:21.982031 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.982035 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqk9k\" (UniqueName: \"kubernetes.io/projected/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kube-api-access-fqk9k\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:09:21.982250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.982047 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dadd5ba7-86a7-45b0-981a-9ff8769e910c-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:09:21.982250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:21.982055 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dadd5ba7-86a7-45b0-981a-9ff8769e910c-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:09:22.114134 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.114098 5902 generic.go:356] "Generic (PLEG): container finished" podID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerID="9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b" exitCode=0 Oct 30 03:09:22.114287 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.114181 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" Oct 30 03:09:22.114287 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.114176 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerDied","Data":"9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b"} Oct 30 03:09:22.114362 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.114333 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl" event={"ID":"dadd5ba7-86a7-45b0-981a-9ff8769e910c","Type":"ContainerDied","Data":"3a60260f8ece7ea2dec3ac6a744e82f13132599f6ab8e4bc7403b1087fa1e5e7"} Oct 30 03:09:22.114395 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.114383 5902 scope.go:122] "RemoveContainer" containerID="9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b" Oct 30 03:09:22.122538 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.122518 5902 scope.go:122] "RemoveContainer" containerID="8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a" Oct 30 03:09:22.129914 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.129888 5902 scope.go:122] "RemoveContainer" containerID="e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55" Oct 30 03:09:22.137309 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.137294 5902 scope.go:122] "RemoveContainer" containerID="f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd" Oct 30 03:09:22.144216 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.144200 5902 scope.go:122] "RemoveContainer" containerID="9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b" Oct 30 03:09:22.144461 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:09:22.144441 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b\": container with ID starting with 9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b not found: ID does not exist" containerID="9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b" Oct 30 03:09:22.144521 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.144469 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b"} err="failed to get container status \"9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b\": rpc error: code = NotFound desc = could not find container \"9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b\": container with ID starting with 9b6e5ab359cc45d4b34f7a4b5c3f9b3b39204c1fe9459be4e06c34dcf662d21b not found: ID does not exist" Oct 30 03:09:22.144521 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.144487 5902 scope.go:122] "RemoveContainer" containerID="8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a" Oct 30 03:09:22.144745 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:09:22.144730 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a\": container with ID starting with 8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a not found: ID does not exist" containerID="8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a" Oct 30 03:09:22.144789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.144748 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a"} err="failed to get container status \"8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a\": rpc error: code = NotFound desc = could not find container \"8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a\": container with ID starting with 8d5e76e1b104faeb05469f26a67fd4d01c2ef974ee1be1e812d00c66ddc6833a not found: ID does not exist" Oct 30 03:09:22.144789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.144762 5902 scope.go:122] "RemoveContainer" containerID="e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55" Oct 30 03:09:22.145063 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:09:22.145045 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55\": container with ID starting with e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55 not found: ID does not exist" containerID="e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55" Oct 30 03:09:22.145100 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.145070 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55"} err="failed to get container status \"e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55\": rpc error: code = NotFound desc = could not find container \"e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55\": container with ID starting with e41bf2ffa741423eccf049d59be376a5f934325f89c24f398a23f7fff9f86a55 not found: ID does not exist" Oct 30 03:09:22.145100 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.145086 5902 scope.go:122] "RemoveContainer" containerID="f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd" Oct 30 03:09:22.145297 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:09:22.145281 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd\": container with ID starting with f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd not found: ID does not exist" containerID="f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd" Oct 30 03:09:22.145348 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.145301 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd"} err="failed to get container status \"f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd\": rpc error: code = NotFound desc = could not find container \"f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd\": container with ID starting with f9e9cc7e417531a28226879590078deabeaa3db9c8f725d22ad45f3c1aee92bd not found: ID does not exist" Oct 30 03:09:22.164973 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.164950 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl"] Oct 30 03:09:22.175457 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:22.175432 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-e6d7890379/isvc-raw-sklearn-batcher-cac05-predictor-78cd477dc9-kq8xl"] Oct 30 03:09:23.017777 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.017733 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" path="/var/lib/kubelet/pods/dadd5ba7-86a7-45b0-981a-9ff8769e910c/volumes" Oct 30 03:09:23.030627 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.030583 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:09:23.845154 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845120 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb"] Oct 30 03:09:23.845448 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845435 5902 cpu_manager.go:414] "containerMap: removing container" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" Oct 30 03:09:23.845496 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845449 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" Oct 30 03:09:23.845496 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845470 5902 cpu_manager.go:414] "containerMap: removing container" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" Oct 30 03:09:23.845496 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845476 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" Oct 30 03:09:23.845496 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845483 5902 cpu_manager.go:414] "containerMap: removing container" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" Oct 30 03:09:23.845496 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845488 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845498 5902 cpu_manager.go:414] "containerMap: removing container" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="storage-initializer" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845503 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="storage-initializer" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845510 5902 cpu_manager.go:414] "containerMap: removing container" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="storage-initializer" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845515 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="storage-initializer" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845520 5902 cpu_manager.go:414] "containerMap: removing container" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845525 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845532 5902 cpu_manager.go:414] "containerMap: removing container" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845537 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845547 5902 cpu_manager.go:414] "containerMap: removing container" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="storage-initializer" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845552 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="storage-initializer" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845564 5902 cpu_manager.go:414] "containerMap: removing container" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845568 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845574 5902 cpu_manager.go:414] "containerMap: removing container" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845578 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845625 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kserve-container" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845634 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kserve-container" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845640 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="kube-rbac-proxy" Oct 30 03:09:23.845645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845645 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kube-rbac-proxy" Oct 30 03:09:23.846354 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845653 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f643df2-d227-4fbd-8140-7c2a47ca7e25" containerName="kube-rbac-proxy" Oct 30 03:09:23.846354 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845659 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="dadd5ba7-86a7-45b0-981a-9ff8769e910c" containerName="agent" Oct 30 03:09:23.846354 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.845664 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="f788ffdc-1b61-438f-9812-41217114b262" containerName="kserve-container" Oct 30 03:09:23.850500 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.850484 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.865254 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.865229 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"seaweedfs-s3-creds\"" Oct 30 03:09:23.875537 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.875496 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"openshift-service-ca.crt\"" Oct 30 03:09:23.875537 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.875512 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"default-dockercfg-f9xch\"" Oct 30 03:09:23.875537 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.875518 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:09:23.875797 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.875551 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"kube-root-ca.crt\"" Oct 30 03:09:23.875797 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.875760 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-xgboost-graph-raw-3299d-predictor-serving-cert\"" Oct 30 03:09:23.875919 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.875809 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-xgboost-graph-raw-3299d-kube-rbac-proxy-sar-config\"" Oct 30 03:09:23.877201 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.877181 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb"] Oct 30 03:09:23.897912 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.897885 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.898010 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.897928 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtbzh\" (UniqueName: \"kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.898010 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.897959 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.898088 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.898052 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.898126 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.898099 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999138 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999104 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999149 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999249 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtbzh\" (UniqueName: \"kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999271 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999295 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999599 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999564 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:23.999982 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999960 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:24.000075 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:23.999966 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:24.001727 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:24.001708 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:24.022600 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:24.022574 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtbzh\" (UniqueName: \"kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh\") pod \"isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:24.168538 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:24.168456 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:24.480886 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:24.480851 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb"] Oct 30 03:09:24.484424 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:09:24.484393 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod906c99b6_229b_4534_9f50_f3845920f17b.slice/crio-bad3c899467d0a6b9c2caea132384fc026f93038b0fcbad367ad970e29874337 WatchSource:0}: Error finding container bad3c899467d0a6b9c2caea132384fc026f93038b0fcbad367ad970e29874337: Status 404 returned error can't find the container with id bad3c899467d0a6b9c2caea132384fc026f93038b0fcbad367ad970e29874337 Oct 30 03:09:25.128497 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:25.128461 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerStarted","Data":"db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399"} Oct 30 03:09:25.128960 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:25.128530 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerStarted","Data":"bad3c899467d0a6b9c2caea132384fc026f93038b0fcbad367ad970e29874337"} Oct 30 03:09:29.145193 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:29.145161 5902 generic.go:356] "Generic (PLEG): container finished" podID="906c99b6-229b-4534-9f50-f3845920f17b" containerID="db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399" exitCode=0 Oct 30 03:09:29.145670 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:29.145233 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerDied","Data":"db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399"} Oct 30 03:09:33.030042 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:33.030000 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:09:41.721650 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:09:41.721609 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:09:43.031055 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:43.030621 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:09:49.232330 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:49.232291 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerStarted","Data":"0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29"} Oct 30 03:09:49.232846 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:49.232353 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerStarted","Data":"05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e"} Oct 30 03:09:49.232846 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:49.232538 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:50.236569 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:50.236538 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:50.237874 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:50.237843 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:09:51.240245 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:51.240200 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:09:53.030510 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:53.030463 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:09:56.245230 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:56.245200 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:09:56.245785 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:56.245752 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:09:56.298587 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:09:56.298542 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podStartSLOduration=10.423289955 podStartE2EDuration="33.298529224s" totalImagesPullingTime="19.098986398s" totalInitContainerRuntime="3.776252871s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:09:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:09:49.296185603 +0000 UTC m=+802.393602162" watchObservedRunningTime="2026-10-30 03:09:56.298529224 +0000 UTC m=+809.395945780" Oct 30 03:10:03.030042 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:03.029990 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:10:06.246007 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:06.245968 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:10:11.719272 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:10:11.719185 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:10:13.030284 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:13.030243 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused" Oct 30 03:10:16.245899 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:16.245861 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:10:19.017792 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:19.017760 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:10:21.306538 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:21.306505 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8"] Oct 30 03:10:21.306968 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:21.306706 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" containerID="cri-o://f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b" gracePeriod=30 Oct 30 03:10:21.306968 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:21.306758 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kube-rbac-proxy" containerID="cri-o://f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116" gracePeriod=30 Oct 30 03:10:22.352292 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:22.352257 5902 generic.go:356] "Generic (PLEG): container finished" podID="57c6d688-d7e7-4839-a790-0de78007fd46" containerID="f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116" exitCode=0 Oct 30 03:10:22.352661 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:22.352299 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerDied","Data":"f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116"} Oct 30 03:10:23.026065 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:23.026026 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused" Oct 30 03:10:26.245792 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.245750 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:10:26.542712 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.542687 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:10:26.667228 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667197 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config\") pod \"57c6d688-d7e7-4839-a790-0de78007fd46\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " Oct 30 03:10:26.667393 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667233 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh\" (UniqueName: \"kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh\") pod \"57c6d688-d7e7-4839-a790-0de78007fd46\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " Oct 30 03:10:26.667393 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667263 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls\" (UniqueName: \"kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls\") pod \"57c6d688-d7e7-4839-a790-0de78007fd46\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " Oct 30 03:10:26.667393 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667348 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location\") pod \"57c6d688-d7e7-4839-a790-0de78007fd46\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " Oct 30 03:10:26.667511 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667412 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert\") pod \"57c6d688-d7e7-4839-a790-0de78007fd46\" (UID: \"57c6d688-d7e7-4839-a790-0de78007fd46\") " Oct 30 03:10:26.667751 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667718 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location" pod "57c6d688-d7e7-4839-a790-0de78007fd46" (UID: "57c6d688-d7e7-4839-a790-0de78007fd46"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:10:26.667751 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667734 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config" pod "57c6d688-d7e7-4839-a790-0de78007fd46" (UID: "57c6d688-d7e7-4839-a790-0de78007fd46"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:10:26.667965 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.667866 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert" pod "57c6d688-d7e7-4839-a790-0de78007fd46" (UID: "57c6d688-d7e7-4839-a790-0de78007fd46"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:10:26.669462 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.669433 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh" pod "57c6d688-d7e7-4839-a790-0de78007fd46" (UID: "57c6d688-d7e7-4839-a790-0de78007fd46"). InnerVolumeSpecName "kube-api-access-gqzxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:10:26.669589 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.669462 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls" pod "57c6d688-d7e7-4839-a790-0de78007fd46" (UID: "57c6d688-d7e7-4839-a790-0de78007fd46"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:10:26.768574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.768532 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:10:26.768574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.768566 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqzxh\" (UniqueName: \"kubernetes.io/projected/57c6d688-d7e7-4839-a790-0de78007fd46-kube-api-access-gqzxh\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:10:26.768574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.768577 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/57c6d688-d7e7-4839-a790-0de78007fd46-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:10:26.768574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.768587 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/57c6d688-d7e7-4839-a790-0de78007fd46-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:10:26.768883 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:26.768595 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/57c6d688-d7e7-4839-a790-0de78007fd46-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:10:27.371255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.371178 5902 generic.go:356] "Generic (PLEG): container finished" podID="57c6d688-d7e7-4839-a790-0de78007fd46" containerID="f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b" exitCode=0 Oct 30 03:10:27.371255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.371213 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerDied","Data":"f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b"} Oct 30 03:10:27.371255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.371235 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" Oct 30 03:10:27.371777 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.371288 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8" event={"ID":"57c6d688-d7e7-4839-a790-0de78007fd46","Type":"ContainerDied","Data":"656b9f88bd494b2a0448615c491d8f62ca9bb121c795e62e0a24fe1f4335b17f"} Oct 30 03:10:27.371777 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.371342 5902 scope.go:122] "RemoveContainer" containerID="f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116" Oct 30 03:10:27.379186 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.379168 5902 scope.go:122] "RemoveContainer" containerID="f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b" Oct 30 03:10:27.386019 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.386000 5902 scope.go:122] "RemoveContainer" containerID="a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25" Oct 30 03:10:27.392686 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.392671 5902 scope.go:122] "RemoveContainer" containerID="f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116" Oct 30 03:10:27.392968 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:10:27.392950 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116\": container with ID starting with f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116 not found: ID does not exist" containerID="f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116" Oct 30 03:10:27.393029 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.392977 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116"} err="failed to get container status \"f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116\": rpc error: code = NotFound desc = could not find container \"f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116\": container with ID starting with f4d93fe3f4038975a602d91c4338c8bb61a492e297c2912b26c0a2cd3fcfa116 not found: ID does not exist" Oct 30 03:10:27.393029 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.392995 5902 scope.go:122] "RemoveContainer" containerID="f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b" Oct 30 03:10:27.393242 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:10:27.393223 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b\": container with ID starting with f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b not found: ID does not exist" containerID="f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b" Oct 30 03:10:27.393311 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.393251 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b"} err="failed to get container status \"f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b\": rpc error: code = NotFound desc = could not find container \"f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b\": container with ID starting with f27bbac7cbfbad6a032424aaeec2558ec67b1b14832dd8b0756fac3923ceb90b not found: ID does not exist" Oct 30 03:10:27.393311 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.393274 5902 scope.go:122] "RemoveContainer" containerID="a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25" Oct 30 03:10:27.393505 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:10:27.393488 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25\": container with ID starting with a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25 not found: ID does not exist" containerID="a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25" Oct 30 03:10:27.393542 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.393510 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25"} err="failed to get container status \"a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25\": rpc error: code = NotFound desc = could not find container \"a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25\": container with ID starting with a78619c0fde54de80a8610226c206495765f8850ec42aab16f8748a2b0b8bb25 not found: ID does not exist" Oct 30 03:10:27.430446 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.430417 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8"] Oct 30 03:10:27.441042 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:27.441017 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-rollback-predictor-598dbf8bc4-mhxr8"] Oct 30 03:10:29.018321 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.018291 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" path="/var/lib/kubelet/pods/57c6d688-d7e7-4839-a790-0de78007fd46/volumes" Oct 30 03:10:29.210876 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.210844 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm"] Oct 30 03:10:29.211148 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211136 5902 cpu_manager.go:414] "containerMap: removing container" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" Oct 30 03:10:29.211190 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211149 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" Oct 30 03:10:29.211190 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211165 5902 cpu_manager.go:414] "containerMap: removing container" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kube-rbac-proxy" Oct 30 03:10:29.211190 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211170 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kube-rbac-proxy" Oct 30 03:10:29.211284 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211203 5902 cpu_manager.go:414] "containerMap: removing container" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="storage-initializer" Oct 30 03:10:29.211284 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211209 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="storage-initializer" Oct 30 03:10:29.211284 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211248 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kube-rbac-proxy" Oct 30 03:10:29.211284 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.211260 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c6d688-d7e7-4839-a790-0de78007fd46" containerName="kserve-container" Oct 30 03:10:29.214330 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.214311 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.228183 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.228162 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"seaweedfs-s3-creds\"" Oct 30 03:10:29.238419 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.238398 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-stop-predictor-serving-cert\"" Oct 30 03:10:29.238520 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.238416 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"default-dockercfg-57rzd\"" Oct 30 03:10:29.239458 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.239436 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm"] Oct 30 03:10:29.267603 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.267571 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"openshift-service-ca.crt\"" Oct 30 03:10:29.267734 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.267643 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\"" Oct 30 03:10:29.267734 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.267664 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"kube-root-ca.crt\"" Oct 30 03:10:29.267734 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.267666 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:10:29.389268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.389173 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.389268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.389223 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.389472 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.389308 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rptrn\" (UniqueName: \"kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.389472 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.389366 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.389472 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.389388 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.490775 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.490736 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rptrn\" (UniqueName: \"kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.490792 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.490813 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.490970 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491239 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.491010 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491353 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.491334 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491542 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.491512 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.491542 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.491522 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.493271 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.493248 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.513638 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.513606 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rptrn\" (UniqueName: \"kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn\") pod \"isvc-canary-stop-predictor-94cccdcc4-b4mzm\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.528283 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.528257 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:29.847902 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:29.847872 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm"] Oct 30 03:10:29.851101 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:10:29.851074 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc113da2_c602_409d_8243_b95dc9b674e9.slice/crio-7b77e738f3d6963fea9a8c71a832bec865d6d2a323e1c9777f719c3e710e6ff3 WatchSource:0}: Error finding container 7b77e738f3d6963fea9a8c71a832bec865d6d2a323e1c9777f719c3e710e6ff3: Status 404 returned error can't find the container with id 7b77e738f3d6963fea9a8c71a832bec865d6d2a323e1c9777f719c3e710e6ff3 Oct 30 03:10:30.389973 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:30.389934 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerStarted","Data":"7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82"} Oct 30 03:10:30.390412 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:30.390021 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerStarted","Data":"7b77e738f3d6963fea9a8c71a832bec865d6d2a323e1c9777f719c3e710e6ff3"} Oct 30 03:10:34.405224 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:34.405189 5902 generic.go:356] "Generic (PLEG): container finished" podID="cc113da2-c602-409d-8243-b95dc9b674e9" containerID="7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82" exitCode=0 Oct 30 03:10:34.405613 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:34.405250 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerDied","Data":"7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82"} Oct 30 03:10:35.410742 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:35.410704 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerStarted","Data":"051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6"} Oct 30 03:10:35.411152 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:35.410772 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerStarted","Data":"eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e"} Oct 30 03:10:35.411152 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:35.410989 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:35.477913 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:35.477864 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podStartSLOduration=2.82310353 podStartE2EDuration="6.477850266s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.654746736s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:10:35.467195247 +0000 UTC m=+848.564611805" watchObservedRunningTime="2026-10-30 03:10:35.477850266 +0000 UTC m=+848.575266819" Oct 30 03:10:36.246385 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:36.246342 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:10:36.414978 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:36.414950 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:36.416300 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:36.416267 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:10:37.418710 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:37.418669 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:10:41.718869 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:10:41.718750 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:10:42.423471 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:42.423432 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:10:42.424070 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:42.424037 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:10:46.246343 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:46.246299 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:10:52.424001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:52.423965 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:10:56.246721 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:10:56.246691 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:11:02.424250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:02.424210 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:11:11.719009 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:11:11.718973 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:11:12.424837 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:12.424784 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:11:14.198494 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:14.198463 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb"] Oct 30 03:11:14.199074 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:14.198715 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" containerID="cri-o://05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e" gracePeriod=30 Oct 30 03:11:14.199074 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:14.198769 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kube-rbac-proxy" containerID="cri-o://0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29" gracePeriod=30 Oct 30 03:11:14.557766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:14.557683 5902 generic.go:356] "Generic (PLEG): container finished" podID="906c99b6-229b-4534-9f50-f3845920f17b" containerID="0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29" exitCode=0 Oct 30 03:11:14.557766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:14.557725 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerDied","Data":"0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29"} Oct 30 03:11:16.240596 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:16.240553 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused" Oct 30 03:11:16.245928 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:16.245903 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused" Oct 30 03:11:18.439262 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.439236 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:11:18.575536 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.575443 5902 generic.go:356] "Generic (PLEG): container finished" podID="906c99b6-229b-4534-9f50-f3845920f17b" containerID="05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e" exitCode=0 Oct 30 03:11:18.575536 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.575515 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" Oct 30 03:11:18.575766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.575527 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerDied","Data":"05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e"} Oct 30 03:11:18.575766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.575603 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb" event={"ID":"906c99b6-229b-4534-9f50-f3845920f17b","Type":"ContainerDied","Data":"bad3c899467d0a6b9c2caea132384fc026f93038b0fcbad367ad970e29874337"} Oct 30 03:11:18.575766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.575639 5902 scope.go:122] "RemoveContainer" containerID="0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29" Oct 30 03:11:18.585337 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585312 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert\") pod \"906c99b6-229b-4534-9f50-f3845920f17b\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " Oct 30 03:11:18.585475 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585366 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location\") pod \"906c99b6-229b-4534-9f50-f3845920f17b\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " Oct 30 03:11:18.585540 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585468 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config\") pod \"906c99b6-229b-4534-9f50-f3845920f17b\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " Oct 30 03:11:18.585612 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585596 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh\" (UniqueName: \"kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh\") pod \"906c99b6-229b-4534-9f50-f3845920f17b\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " Oct 30 03:11:18.585731 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585649 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location" pod "906c99b6-229b-4534-9f50-f3845920f17b" (UID: "906c99b6-229b-4534-9f50-f3845920f17b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:11:18.585916 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585665 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls\" (UniqueName: \"kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls\") pod \"906c99b6-229b-4534-9f50-f3845920f17b\" (UID: \"906c99b6-229b-4534-9f50-f3845920f17b\") " Oct 30 03:11:18.585916 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.585744 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert" pod "906c99b6-229b-4534-9f50-f3845920f17b" (UID: "906c99b6-229b-4534-9f50-f3845920f17b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:11:18.586033 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.586003 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/906c99b6-229b-4534-9f50-f3845920f17b-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:11:18.586033 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.586023 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:11:18.586168 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.586096 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config" pod "906c99b6-229b-4534-9f50-f3845920f17b" (UID: "906c99b6-229b-4534-9f50-f3845920f17b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:11:18.586285 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.586267 5902 scope.go:122] "RemoveContainer" containerID="05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e" Oct 30 03:11:18.588214 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.588190 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh" pod "906c99b6-229b-4534-9f50-f3845920f17b" (UID: "906c99b6-229b-4534-9f50-f3845920f17b"). InnerVolumeSpecName "kube-api-access-jtbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:11:18.588357 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.588338 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls" pod "906c99b6-229b-4534-9f50-f3845920f17b" (UID: "906c99b6-229b-4534-9f50-f3845920f17b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:11:18.602123 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.602108 5902 scope.go:122] "RemoveContainer" containerID="db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399" Oct 30 03:11:18.609422 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.609407 5902 scope.go:122] "RemoveContainer" containerID="0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29" Oct 30 03:11:18.609708 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:11:18.609689 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29\": container with ID starting with 0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29 not found: ID does not exist" containerID="0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29" Oct 30 03:11:18.609757 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.609718 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29"} err="failed to get container status \"0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29\": rpc error: code = NotFound desc = could not find container \"0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29\": container with ID starting with 0a3827ed76116452d0dd3d5ef0beaf32260cd6ea08e7a04f1d26759d7a19ac29 not found: ID does not exist" Oct 30 03:11:18.609757 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.609735 5902 scope.go:122] "RemoveContainer" containerID="05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e" Oct 30 03:11:18.609984 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:11:18.609967 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e\": container with ID starting with 05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e not found: ID does not exist" containerID="05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e" Oct 30 03:11:18.610025 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.609988 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e"} err="failed to get container status \"05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e\": rpc error: code = NotFound desc = could not find container \"05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e\": container with ID starting with 05b8963422ae6009cc32b4dd9750806e1e15aaa06a2faf16a48b8fec6d3e3e5e not found: ID does not exist" Oct 30 03:11:18.610025 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.610003 5902 scope.go:122] "RemoveContainer" containerID="db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399" Oct 30 03:11:18.610213 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:11:18.610198 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399\": container with ID starting with db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399 not found: ID does not exist" containerID="db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399" Oct 30 03:11:18.610269 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.610217 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399"} err="failed to get container status \"db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399\": rpc error: code = NotFound desc = could not find container \"db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399\": container with ID starting with db9441038252daa1b06273e9351ffaa543c4af414fd4054dabb03e90fc13c399 not found: ID does not exist" Oct 30 03:11:18.686714 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.686665 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/906c99b6-229b-4534-9f50-f3845920f17b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:11:18.686714 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.686709 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtbzh\" (UniqueName: \"kubernetes.io/projected/906c99b6-229b-4534-9f50-f3845920f17b-kube-api-access-jtbzh\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:11:18.686714 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.686722 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/906c99b6-229b-4534-9f50-f3845920f17b-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:11:18.918809 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.918774 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb"] Oct 30 03:11:18.929937 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:18.929907 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-e6d7890379/isvc-xgboost-graph-raw-3299d-predictor-7bd855f9ff-rqjmb"] Oct 30 03:11:19.022276 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:19.020341 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="906c99b6-229b-4534-9f50-f3845920f17b" path="/var/lib/kubelet/pods/906c99b6-229b-4534-9f50-f3845920f17b/volumes" Oct 30 03:11:21.652695 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.652660 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h"] Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653001 5902 cpu_manager.go:414] "containerMap: removing container" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kube-rbac-proxy" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653018 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kube-rbac-proxy" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653030 5902 cpu_manager.go:414] "containerMap: removing container" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653035 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653046 5902 cpu_manager.go:414] "containerMap: removing container" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="storage-initializer" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653050 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="storage-initializer" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653133 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kube-rbac-proxy" Oct 30 03:11:21.653171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.653141 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="906c99b6-229b-4534-9f50-f3845920f17b" containerName="kserve-container" Oct 30 03:11:21.657402 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.657380 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.683891 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.683863 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"openshift-service-ca.crt\"" Oct 30 03:11:21.683891 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.683862 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:11:21.684183 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.683938 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"kube-root-ca.crt\"" Oct 30 03:11:21.684183 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.683952 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"seaweedfs-s3-creds\"" Oct 30 03:11:21.684183 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.683989 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-sklearn-graph-raw-hpa-fdaf1-kube-rbac-proxy-sar-config\"" Oct 30 03:11:21.684183 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.684024 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-serving-cert\"" Oct 30 03:11:21.684183 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.684069 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"default-dockercfg-f9xch\"" Oct 30 03:11:21.685881 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.685861 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h"] Oct 30 03:11:21.815191 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.815153 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.815367 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.815200 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjwc\" (UniqueName: \"kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.815367 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.815290 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.815437 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.815387 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.815437 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.815409 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.916529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.916415 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.916529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.916469 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjwc\" (UniqueName: \"kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.916529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.916499 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.916529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.916527 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.916846 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.916544 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.917030 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.917010 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.917150 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.917130 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.917208 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.917191 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.918971 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.918953 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.939213 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.939182 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjwc\" (UniqueName: \"kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc\") pod \"isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:21.973323 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:21.973294 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:22.285462 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:22.285426 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h"] Oct 30 03:11:22.289025 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:11:22.288986 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc38c0bad_602a_4413_8d63_3e7cdb32bbe3.slice/crio-0cfc689753ead6ecc14f6bf7fee62203f89d06beefe05872f9ee1b0e0d643790 WatchSource:0}: Error finding container 0cfc689753ead6ecc14f6bf7fee62203f89d06beefe05872f9ee1b0e0d643790: Status 404 returned error can't find the container with id 0cfc689753ead6ecc14f6bf7fee62203f89d06beefe05872f9ee1b0e0d643790 Oct 30 03:11:22.424956 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:22.424914 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:11:22.593948 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:22.593863 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerStarted","Data":"aeabd04cea10058b152f6787e3b47d3dd9388e1d793f1691f5b2715d028d1822"} Oct 30 03:11:22.593948 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:22.593930 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerStarted","Data":"0cfc689753ead6ecc14f6bf7fee62203f89d06beefe05872f9ee1b0e0d643790"} Oct 30 03:11:26.610559 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:26.610518 5902 generic.go:356] "Generic (PLEG): container finished" podID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerID="aeabd04cea10058b152f6787e3b47d3dd9388e1d793f1691f5b2715d028d1822" exitCode=0 Oct 30 03:11:26.610559 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:26.610559 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerDied","Data":"aeabd04cea10058b152f6787e3b47d3dd9388e1d793f1691f5b2715d028d1822"} Oct 30 03:11:27.617018 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:27.616983 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerStarted","Data":"fc6d42711f2c8afd8cd3ddf0e4b9534ddc150d1e51a72bf289278d287d62877b"} Oct 30 03:11:27.617420 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:27.617067 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerStarted","Data":"b431262727e46901627df1b42d2a8767f38aee4d3110b93d2f9454a6f7a255c7"} Oct 30 03:11:27.617420 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:27.617344 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:27.617420 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:27.617389 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:27.618638 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:27.618613 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:11:27.674513 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:27.674437 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podStartSLOduration=2.8876282509999998 podStartE2EDuration="6.67441836s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786790109s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:11:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:11:27.673849604 +0000 UTC m=+900.771266171" watchObservedRunningTime="2026-10-30 03:11:27.67441836 +0000 UTC m=+900.771834921" Oct 30 03:11:28.621028 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:28.620980 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:11:32.424503 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:32.424460 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:11:33.625999 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:33.625968 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:11:33.626660 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:33.626629 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:11:41.719584 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:11:41.719546 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:11:42.424030 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:42.423993 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:11:43.627396 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:43.627349 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:11:52.425017 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:52.424985 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:11:53.627098 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:11:53.627063 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:12:03.627464 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:03.627428 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:12:09.291075 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:09.291040 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm"] Oct 30 03:12:09.291566 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:09.291321 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" containerID="cri-o://eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e" gracePeriod=30 Oct 30 03:12:09.291566 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:09.291407 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kube-rbac-proxy" containerID="cri-o://051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6" gracePeriod=30 Oct 30 03:12:09.777763 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:09.777730 5902 generic.go:356] "Generic (PLEG): container finished" podID="cc113da2-c602-409d-8243-b95dc9b674e9" containerID="051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6" exitCode=0 Oct 30 03:12:09.777971 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:09.777808 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerDied","Data":"051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6"} Oct 30 03:12:11.719215 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:12:11.719180 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:12:12.419503 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:12.419456 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused" Oct 30 03:12:12.424233 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:12.424207 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused" Oct 30 03:12:13.626973 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:13.626932 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:12:14.435806 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.435780 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:12:14.472680 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.472652 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls\") pod \"cc113da2-c602-409d-8243-b95dc9b674e9\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " Oct 30 03:12:14.472845 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.472688 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert\") pod \"cc113da2-c602-409d-8243-b95dc9b674e9\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " Oct 30 03:12:14.472845 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.472727 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config\") pod \"cc113da2-c602-409d-8243-b95dc9b674e9\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " Oct 30 03:12:14.472845 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.472755 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn\" (UniqueName: \"kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn\") pod \"cc113da2-c602-409d-8243-b95dc9b674e9\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " Oct 30 03:12:14.472845 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.472784 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location\") pod \"cc113da2-c602-409d-8243-b95dc9b674e9\" (UID: \"cc113da2-c602-409d-8243-b95dc9b674e9\") " Oct 30 03:12:14.473162 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.473137 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location" pod "cc113da2-c602-409d-8243-b95dc9b674e9" (UID: "cc113da2-c602-409d-8243-b95dc9b674e9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:12:14.473162 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.473147 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config" pod "cc113da2-c602-409d-8243-b95dc9b674e9" (UID: "cc113da2-c602-409d-8243-b95dc9b674e9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:14.473278 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.473151 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert" pod "cc113da2-c602-409d-8243-b95dc9b674e9" (UID: "cc113da2-c602-409d-8243-b95dc9b674e9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:12:14.474866 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.474839 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls" pod "cc113da2-c602-409d-8243-b95dc9b674e9" (UID: "cc113da2-c602-409d-8243-b95dc9b674e9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:12:14.474946 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.474927 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn" pod "cc113da2-c602-409d-8243-b95dc9b674e9" (UID: "cc113da2-c602-409d-8243-b95dc9b674e9"). InnerVolumeSpecName "kube-api-access-rptrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:12:14.573859 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.573758 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cc113da2-c602-409d-8243-b95dc9b674e9-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:12:14.573859 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.573785 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:12:14.573859 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.573796 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cc113da2-c602-409d-8243-b95dc9b674e9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:12:14.573859 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.573805 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rptrn\" (UniqueName: \"kubernetes.io/projected/cc113da2-c602-409d-8243-b95dc9b674e9-kube-api-access-rptrn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:12:14.573859 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.573844 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cc113da2-c602-409d-8243-b95dc9b674e9-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:12:14.796683 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.796648 5902 generic.go:356] "Generic (PLEG): container finished" podID="cc113da2-c602-409d-8243-b95dc9b674e9" containerID="eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e" exitCode=0 Oct 30 03:12:14.797087 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.796724 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" Oct 30 03:12:14.797087 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.796732 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerDied","Data":"eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e"} Oct 30 03:12:14.797087 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.796794 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm" event={"ID":"cc113da2-c602-409d-8243-b95dc9b674e9","Type":"ContainerDied","Data":"7b77e738f3d6963fea9a8c71a832bec865d6d2a323e1c9777f719c3e710e6ff3"} Oct 30 03:12:14.797087 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.796846 5902 scope.go:122] "RemoveContainer" containerID="051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6" Oct 30 03:12:14.804919 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.804903 5902 scope.go:122] "RemoveContainer" containerID="eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e" Oct 30 03:12:14.812142 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.812124 5902 scope.go:122] "RemoveContainer" containerID="7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82" Oct 30 03:12:14.819621 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.819585 5902 scope.go:122] "RemoveContainer" containerID="051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6" Oct 30 03:12:14.819946 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:12:14.819926 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6\": container with ID starting with 051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6 not found: ID does not exist" containerID="051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6" Oct 30 03:12:14.820008 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.819958 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6"} err="failed to get container status \"051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6\": rpc error: code = NotFound desc = could not find container \"051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6\": container with ID starting with 051c1f7f6d8102c52b9cda1ab6780b643a9ba1818a703e3d461c9481392b52a6 not found: ID does not exist" Oct 30 03:12:14.820008 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.819978 5902 scope.go:122] "RemoveContainer" containerID="eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e" Oct 30 03:12:14.820260 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:12:14.820243 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e\": container with ID starting with eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e not found: ID does not exist" containerID="eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e" Oct 30 03:12:14.820304 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.820280 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e"} err="failed to get container status \"eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e\": rpc error: code = NotFound desc = could not find container \"eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e\": container with ID starting with eaf24133ee7e1dce93ec10ec66b84f0d562bb175200a37983941d5194bb9329e not found: ID does not exist" Oct 30 03:12:14.820304 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.820298 5902 scope.go:122] "RemoveContainer" containerID="7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82" Oct 30 03:12:14.820568 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:12:14.820549 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82\": container with ID starting with 7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82 not found: ID does not exist" containerID="7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82" Oct 30 03:12:14.820622 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.820577 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82"} err="failed to get container status \"7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82\": rpc error: code = NotFound desc = could not find container \"7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82\": container with ID starting with 7c336593a99ada1b529f4ef1d901ea15e6882947aa57803700a07222bd161c82 not found: ID does not exist" Oct 30 03:12:14.844934 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.844860 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm"] Oct 30 03:12:14.855401 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:14.855379 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-stop-predictor-94cccdcc4-b4mzm"] Oct 30 03:12:15.023142 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:15.023107 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" path="/var/lib/kubelet/pods/cc113da2-c602-409d-8243-b95dc9b674e9/volumes" Oct 30 03:12:23.626950 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:23.626908 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:12:33.627390 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:33.627342 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:12:38.014414 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:38.014368 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused" Oct 30 03:12:41.719186 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:12:41.719149 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:12:48.014922 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:12:48.014884 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:13:11.719679 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:13:11.719589 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:13:12.278324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:12.278295 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h"] Oct 30 03:13:12.278590 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:12.278542 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" containerID="cri-o://b431262727e46901627df1b42d2a8767f38aee4d3110b93d2f9454a6f7a255c7" gracePeriod=30 Oct 30 03:13:12.278672 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:12.278580 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kube-rbac-proxy" containerID="cri-o://fc6d42711f2c8afd8cd3ddf0e4b9534ddc150d1e51a72bf289278d287d62877b" gracePeriod=30 Oct 30 03:13:13.007223 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:13.007191 5902 generic.go:356] "Generic (PLEG): container finished" podID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerID="fc6d42711f2c8afd8cd3ddf0e4b9534ddc150d1e51a72bf289278d287d62877b" exitCode=0 Oct 30 03:13:13.007608 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:13.007247 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerDied","Data":"fc6d42711f2c8afd8cd3ddf0e4b9534ddc150d1e51a72bf289278d287d62877b"} Oct 30 03:13:13.621357 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:13.621314 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused" Oct 30 03:13:17.022924 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.022892 5902 generic.go:356] "Generic (PLEG): container finished" podID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerID="b431262727e46901627df1b42d2a8767f38aee4d3110b93d2f9454a6f7a255c7" exitCode=0 Oct 30 03:13:17.023392 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.022962 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerDied","Data":"b431262727e46901627df1b42d2a8767f38aee4d3110b93d2f9454a6f7a255c7"} Oct 30 03:13:17.129536 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.129509 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:13:17.281676 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.281645 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls\") pod \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " Oct 30 03:13:17.281888 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.281710 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config\") pod \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " Oct 30 03:13:17.281888 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.281737 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location\") pod \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " Oct 30 03:13:17.281888 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.281768 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc\" (UniqueName: \"kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc\") pod \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " Oct 30 03:13:17.281888 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.281842 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert\") pod \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\" (UID: \"c38c0bad-602a-4413-8d63-3e7cdb32bbe3\") " Oct 30 03:13:17.282171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.282142 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location" pod "c38c0bad-602a-4413-8d63-3e7cdb32bbe3" (UID: "c38c0bad-602a-4413-8d63-3e7cdb32bbe3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:13:17.282456 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.282393 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert" pod "c38c0bad-602a-4413-8d63-3e7cdb32bbe3" (UID: "c38c0bad-602a-4413-8d63-3e7cdb32bbe3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:13:17.282456 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.282411 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config" pod "c38c0bad-602a-4413-8d63-3e7cdb32bbe3" (UID: "c38c0bad-602a-4413-8d63-3e7cdb32bbe3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:13:17.282608 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.282463 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:13:17.284077 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.284057 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls" pod "c38c0bad-602a-4413-8d63-3e7cdb32bbe3" (UID: "c38c0bad-602a-4413-8d63-3e7cdb32bbe3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:13:17.284387 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.284364 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc" pod "c38c0bad-602a-4413-8d63-3e7cdb32bbe3" (UID: "c38c0bad-602a-4413-8d63-3e7cdb32bbe3"). InnerVolumeSpecName "kube-api-access-6rjwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:13:17.383305 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.383271 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rjwc\" (UniqueName: \"kubernetes.io/projected/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-api-access-6rjwc\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:13:17.383305 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.383300 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:13:17.383305 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.383313 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:13:17.383558 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:17.383325 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c38c0bad-602a-4413-8d63-3e7cdb32bbe3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:13:18.028870 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.028836 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" event={"ID":"c38c0bad-602a-4413-8d63-3e7cdb32bbe3","Type":"ContainerDied","Data":"0cfc689753ead6ecc14f6bf7fee62203f89d06beefe05872f9ee1b0e0d643790"} Oct 30 03:13:18.028870 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.028867 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h" Oct 30 03:13:18.029390 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.028914 5902 scope.go:122] "RemoveContainer" containerID="fc6d42711f2c8afd8cd3ddf0e4b9534ddc150d1e51a72bf289278d287d62877b" Oct 30 03:13:18.038490 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.038458 5902 scope.go:122] "RemoveContainer" containerID="b431262727e46901627df1b42d2a8767f38aee4d3110b93d2f9454a6f7a255c7" Oct 30 03:13:18.045727 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.045708 5902 scope.go:122] "RemoveContainer" containerID="aeabd04cea10058b152f6787e3b47d3dd9388e1d793f1691f5b2715d028d1822" Oct 30 03:13:18.089419 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.089384 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h"] Oct 30 03:13:18.099881 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:18.099858 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-e6d7890379/isvc-sklearn-graph-raw-hpa-fdaf1-predictor-67dc66465f-pb46h"] Oct 30 03:13:19.018349 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.018315 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" path="/var/lib/kubelet/pods/c38c0bad-602a-4413-8d63-3e7cdb32bbe3/volumes" Oct 30 03:13:19.814955 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.814922 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j"] Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815220 5902 cpu_manager.go:414] "containerMap: removing container" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815244 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815258 5902 cpu_manager.go:414] "containerMap: removing container" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="storage-initializer" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815275 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="storage-initializer" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815285 5902 cpu_manager.go:414] "containerMap: removing container" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kube-rbac-proxy" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815290 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kube-rbac-proxy" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815318 5902 cpu_manager.go:414] "containerMap: removing container" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" Oct 30 03:13:19.815324 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815323 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815332 5902 cpu_manager.go:414] "containerMap: removing container" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kube-rbac-proxy" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815338 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kube-rbac-proxy" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815352 5902 cpu_manager.go:414] "containerMap: removing container" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="storage-initializer" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815361 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="storage-initializer" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815415 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kserve-container" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815426 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc113da2-c602-409d-8243-b95dc9b674e9" containerName="kube-rbac-proxy" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815436 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kserve-container" Oct 30 03:13:19.815575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.815442 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="c38c0bad-602a-4413-8d63-3e7cdb32bbe3" containerName="kube-rbac-proxy" Oct 30 03:13:19.820048 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.820029 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:19.834769 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.834747 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"message-dumper-raw-27bd3-kube-rbac-proxy-sar-config\"" Oct 30 03:13:19.838672 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.838653 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"openshift-service-ca.crt\"" Oct 30 03:13:19.845537 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.845512 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"message-dumper-raw-27bd3-predictor-serving-cert\"" Oct 30 03:13:19.850148 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.850121 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j"] Oct 30 03:13:19.935280 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.935250 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"kube-root-ca.crt\"" Oct 30 03:13:19.938630 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:19.938615 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"default-dockercfg-f9xch\"" Oct 30 03:13:20.003825 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.003790 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.003943 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.003927 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.003985 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.003958 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbmvw\" (UniqueName: \"kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.105222 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.105144 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.105222 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.105185 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbmvw\" (UniqueName: \"kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.105414 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.105238 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.105936 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.105908 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.107552 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.107534 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.128151 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.128130 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbmvw\" (UniqueName: \"kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw\") pod \"message-dumper-raw-27bd3-predictor-8656945556-7s24j\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.134013 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.133982 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:20.432108 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.432026 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j"] Oct 30 03:13:20.435858 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:13:20.435805 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb795fb8b_9080_43de_9df3_9e359d6bfd32.slice/crio-f63c0ba7a790d1888750f3556bceedd4ba29b4fa2f3f8a2cfbeb9c2888bb5d7b WatchSource:0}: Error finding container f63c0ba7a790d1888750f3556bceedd4ba29b4fa2f3f8a2cfbeb9c2888bb5d7b: Status 404 returned error can't find the container with id f63c0ba7a790d1888750f3556bceedd4ba29b4fa2f3f8a2cfbeb9c2888bb5d7b Oct 30 03:13:20.437706 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:20.437690 5902 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:13:21.040729 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:21.040694 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" event={"ID":"b795fb8b-9080-43de-9df3-9e359d6bfd32","Type":"ContainerStarted","Data":"f63c0ba7a790d1888750f3556bceedd4ba29b4fa2f3f8a2cfbeb9c2888bb5d7b"} Oct 30 03:13:22.046574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:22.046480 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" event={"ID":"b795fb8b-9080-43de-9df3-9e359d6bfd32","Type":"ContainerStarted","Data":"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616"} Oct 30 03:13:22.046574 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:22.046541 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" event={"ID":"b795fb8b-9080-43de-9df3-9e359d6bfd32","Type":"ContainerStarted","Data":"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e"} Oct 30 03:13:22.047094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:22.046641 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:22.090832 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:22.090781 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" podStartSLOduration=1.749258336 podStartE2EDuration="3.090768198s" totalImagesPullingTime="1.341509862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:22.088637422 +0000 UTC m=+1015.186053991" watchObservedRunningTime="2026-10-30 03:13:22.090768198 +0000 UTC m=+1015.188184734" Oct 30 03:13:23.050529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:23.050495 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:23.052307 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:23.052285 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:30.062337 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:30.062302 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:13:40.043600 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.043562 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5"] Oct 30 03:13:40.048564 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.048543 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.062719 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.062692 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"seaweedfs-s3-creds\"" Oct 30 03:13:40.062719 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.062705 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-logger-raw-27bd3-predictor-serving-cert\"" Oct 30 03:13:40.072998 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.072964 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:13:40.072998 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.072964 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-e6d7890379\"/\"isvc-logger-raw-27bd3-kube-rbac-proxy-sar-config\"" Oct 30 03:13:40.073979 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.073958 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5"] Oct 30 03:13:40.166990 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.166960 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.167173 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.167000 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.167173 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.167031 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.167173 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.167161 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.167365 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.167182 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.268501 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.268459 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.268706 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.268508 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.268706 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.268565 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.268706 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.268603 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.268706 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.268640 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.268990 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.268966 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.269229 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.269210 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.269282 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.269250 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.270969 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.270953 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.291710 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.291689 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw\") pod \"isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.367781 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.367703 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:40.690681 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:40.690645 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5"] Oct 30 03:13:40.694121 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:13:40.694097 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56158aa7_e84a_405f_b077_19bb9055f976.slice/crio-70163823bc679ecc8f806082cc6c22cfa46c2d7070bddeca168bea0d98a40944 WatchSource:0}: Error finding container 70163823bc679ecc8f806082cc6c22cfa46c2d7070bddeca168bea0d98a40944: Status 404 returned error can't find the container with id 70163823bc679ecc8f806082cc6c22cfa46c2d7070bddeca168bea0d98a40944 Oct 30 03:13:41.114339 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:41.114255 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerStarted","Data":"8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac"} Oct 30 03:13:41.114339 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:41.114318 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerStarted","Data":"70163823bc679ecc8f806082cc6c22cfa46c2d7070bddeca168bea0d98a40944"} Oct 30 03:13:41.719718 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:13:41.719680 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:13:45.131152 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:45.131118 5902 generic.go:356] "Generic (PLEG): container finished" podID="56158aa7-e84a-405f-b077-19bb9055f976" containerID="8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac" exitCode=0 Oct 30 03:13:45.131689 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:45.131196 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerDied","Data":"8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac"} Oct 30 03:13:46.138354 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.138319 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerStarted","Data":"d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49"} Oct 30 03:13:46.138791 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.138386 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerStarted","Data":"90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593"} Oct 30 03:13:46.138791 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.138412 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerStarted","Data":"07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451"} Oct 30 03:13:46.138791 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.138716 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:46.138791 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.138786 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:46.138972 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.138813 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:46.140142 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.140117 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:13:46.140724 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.140704 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:13:46.204281 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:46.204227 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podStartSLOduration=2.362213687 podStartE2EDuration="6.204213476s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.841999789s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:13:46.203636953 +0000 UTC m=+1039.301053512" watchObservedRunningTime="2026-10-30 03:13:46.204213476 +0000 UTC m=+1039.301630096" Oct 30 03:13:47.143069 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:47.143030 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:13:47.143528 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:47.143396 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:13:52.147546 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:52.147513 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:13:52.148105 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:52.148066 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:13:52.148382 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:13:52.148342 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:02.148403 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:02.148354 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:02.148848 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:02.148806 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:11.719621 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:14:11.719587 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:14:12.148858 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:12.148761 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:12.149205 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:12.149183 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:22.148439 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:22.148400 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:22.148995 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:22.148906 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:32.148042 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:32.148002 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:32.148550 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:32.148513 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:41.719494 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:14:41.719458 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:14:42.148909 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:42.148796 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:42.149298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:42.149271 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:52.148906 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:52.148859 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:52.149413 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:52.149311 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:14:54.014118 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:54.014075 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:14:54.014709 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:14:54.014419 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:15:04.015065 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:04.015028 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:04.015517 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:04.015357 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:11.721244 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:11.721191 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:15:15.304274 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:15.304244 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-e6d7890379_message-dumper-raw-27bd3-predictor-8656945556-7s24j_b795fb8b-9080-43de-9df3-9e359d6bfd32/kserve-container/0.log" Oct 30 03:15:15.913074 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:15.913040 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5"] Oct 30 03:15:15.913367 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:15.913320 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" containerID="cri-o://07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451" gracePeriod=30 Oct 30 03:15:15.913473 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:15.913360 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" containerID="cri-o://d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49" gracePeriod=30 Oct 30 03:15:15.913473 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:15.913370 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" containerID="cri-o://90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593" gracePeriod=30 Oct 30 03:15:16.018016 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.017983 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j"] Oct 30 03:15:16.018238 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.018212 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kserve-container" containerID="cri-o://f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e" gracePeriod=30 Oct 30 03:15:16.018354 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.018271 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kube-rbac-proxy" containerID="cri-o://d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616" gracePeriod=30 Oct 30 03:15:16.253340 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.253316 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:15:16.358854 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.358803 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw\" (UniqueName: \"kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw\") pod \"b795fb8b-9080-43de-9df3-9e359d6bfd32\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " Oct 30 03:15:16.359300 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.358878 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls\" (UniqueName: \"kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls\") pod \"b795fb8b-9080-43de-9df3-9e359d6bfd32\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " Oct 30 03:15:16.359300 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.358927 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config\") pod \"b795fb8b-9080-43de-9df3-9e359d6bfd32\" (UID: \"b795fb8b-9080-43de-9df3-9e359d6bfd32\") " Oct 30 03:15:16.359372 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.359321 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config" pod "b795fb8b-9080-43de-9df3-9e359d6bfd32" (UID: "b795fb8b-9080-43de-9df3-9e359d6bfd32"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:15:16.360934 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.360912 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls" pod "b795fb8b-9080-43de-9df3-9e359d6bfd32" (UID: "b795fb8b-9080-43de-9df3-9e359d6bfd32"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:15:16.361001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.360949 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw" pod "b795fb8b-9080-43de-9df3-9e359d6bfd32" (UID: "b795fb8b-9080-43de-9df3-9e359d6bfd32"). InnerVolumeSpecName "kube-api-access-zbmvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:15:16.460214 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.460184 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbmvw\" (UniqueName: \"kubernetes.io/projected/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-api-access-zbmvw\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.460214 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.460215 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b795fb8b-9080-43de-9df3-9e359d6bfd32-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.460401 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.460224 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b795fb8b-9080-43de-9df3-9e359d6bfd32-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:16.466660 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.466629 5902 generic.go:356] "Generic (PLEG): container finished" podID="56158aa7-e84a-405f-b077-19bb9055f976" containerID="90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593" exitCode=0 Oct 30 03:15:16.466795 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.466699 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerDied","Data":"90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593"} Oct 30 03:15:16.468393 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468363 5902 generic.go:356] "Generic (PLEG): container finished" podID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerID="d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616" exitCode=0 Oct 30 03:15:16.468393 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468381 5902 generic.go:356] "Generic (PLEG): container finished" podID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerID="f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e" exitCode=2 Oct 30 03:15:16.468557 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468410 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" event={"ID":"b795fb8b-9080-43de-9df3-9e359d6bfd32","Type":"ContainerDied","Data":"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616"} Oct 30 03:15:16.468557 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468418 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" Oct 30 03:15:16.468557 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468449 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" event={"ID":"b795fb8b-9080-43de-9df3-9e359d6bfd32","Type":"ContainerDied","Data":"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e"} Oct 30 03:15:16.468557 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468468 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j" event={"ID":"b795fb8b-9080-43de-9df3-9e359d6bfd32","Type":"ContainerDied","Data":"f63c0ba7a790d1888750f3556bceedd4ba29b4fa2f3f8a2cfbeb9c2888bb5d7b"} Oct 30 03:15:16.468557 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.468490 5902 scope.go:122] "RemoveContainer" containerID="d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616" Oct 30 03:15:16.476974 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.476957 5902 scope.go:122] "RemoveContainer" containerID="f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e" Oct 30 03:15:16.484374 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.484361 5902 scope.go:122] "RemoveContainer" containerID="d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616" Oct 30 03:15:16.484634 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:16.484614 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616\": container with ID starting with d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616 not found: ID does not exist" containerID="d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616" Oct 30 03:15:16.484671 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.484642 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616"} err="failed to get container status \"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616\": rpc error: code = NotFound desc = could not find container \"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616\": container with ID starting with d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616 not found: ID does not exist" Oct 30 03:15:16.484671 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.484657 5902 scope.go:122] "RemoveContainer" containerID="f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e" Oct 30 03:15:16.484889 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:16.484872 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e\": container with ID starting with f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e not found: ID does not exist" containerID="f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e" Oct 30 03:15:16.484930 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.484895 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e"} err="failed to get container status \"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e\": rpc error: code = NotFound desc = could not find container \"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e\": container with ID starting with f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e not found: ID does not exist" Oct 30 03:15:16.484930 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.484910 5902 scope.go:122] "RemoveContainer" containerID="d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616" Oct 30 03:15:16.485140 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.485123 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616"} err="failed to get container status \"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616\": rpc error: code = NotFound desc = could not find container \"d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616\": container with ID starting with d26464bfb3c7cbd1558d6ce1571f55d2ea43acd0de6431e406f778d69fe9c616 not found: ID does not exist" Oct 30 03:15:16.485191 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.485141 5902 scope.go:122] "RemoveContainer" containerID="f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e" Oct 30 03:15:16.485347 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.485331 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e"} err="failed to get container status \"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e\": rpc error: code = NotFound desc = could not find container \"f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e\": container with ID starting with f2d686ee0f8a3c407ecc5cfc17f3cfbd03583a5bfba4ad08b55c53e9381d601e not found: ID does not exist" Oct 30 03:15:16.516769 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.516745 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j"] Oct 30 03:15:16.523278 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:16.523258 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-e6d7890379/message-dumper-raw-27bd3-predictor-8656945556-7s24j"] Oct 30 03:15:17.017957 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:17.017851 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" path="/var/lib/kubelet/pods/b795fb8b-9080-43de-9df3-9e359d6bfd32/volumes" Oct 30 03:15:17.144051 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:17.144009 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 30 03:15:21.490555 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:21.490519 5902 generic.go:356] "Generic (PLEG): container finished" podID="56158aa7-e84a-405f-b077-19bb9055f976" containerID="07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451" exitCode=0 Oct 30 03:15:21.490984 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:21.490595 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerDied","Data":"07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451"} Oct 30 03:15:22.143682 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:22.143645 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 30 03:15:24.014039 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:24.013996 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:15:24.014484 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:24.014313 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:15:27.143491 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:27.143452 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 30 03:15:27.144051 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:27.143601 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:32.143422 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:32.143375 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 30 03:15:34.014107 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:34.014067 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:15:34.014504 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:34.014381 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:15:37.143587 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:37.143545 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 30 03:15:41.719953 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:41.719832 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:15:42.143631 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:42.143540 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused" Oct 30 03:15:44.014834 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:44.014774 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused" Oct 30 03:15:44.015292 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:44.014936 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:44.015292 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:44.015164 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:15:44.015292 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:44.015263 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:46.064065 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.064042 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:46.213029 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.212993 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert\") pod \"56158aa7-e84a-405f-b077-19bb9055f976\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " Oct 30 03:15:46.213204 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213046 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location\") pod \"56158aa7-e84a-405f-b077-19bb9055f976\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " Oct 30 03:15:46.213204 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213075 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw\") pod \"56158aa7-e84a-405f-b077-19bb9055f976\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " Oct 30 03:15:46.213204 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213112 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls\" (UniqueName: \"kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls\") pod \"56158aa7-e84a-405f-b077-19bb9055f976\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " Oct 30 03:15:46.213204 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213144 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config\") pod \"56158aa7-e84a-405f-b077-19bb9055f976\" (UID: \"56158aa7-e84a-405f-b077-19bb9055f976\") " Oct 30 03:15:46.213441 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213411 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location" pod "56158aa7-e84a-405f-b077-19bb9055f976" (UID: "56158aa7-e84a-405f-b077-19bb9055f976"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:15:46.213549 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213441 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert" pod "56158aa7-e84a-405f-b077-19bb9055f976" (UID: "56158aa7-e84a-405f-b077-19bb9055f976"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:15:46.213651 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.213628 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config" pod "56158aa7-e84a-405f-b077-19bb9055f976" (UID: "56158aa7-e84a-405f-b077-19bb9055f976"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:15:46.215384 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.215365 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw" pod "56158aa7-e84a-405f-b077-19bb9055f976" (UID: "56158aa7-e84a-405f-b077-19bb9055f976"). InnerVolumeSpecName "kube-api-access-b77gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:15:46.215491 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.215387 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls" pod "56158aa7-e84a-405f-b077-19bb9055f976" (UID: "56158aa7-e84a-405f-b077-19bb9055f976"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:15:46.314610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.314571 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56158aa7-e84a-405f-b077-19bb9055f976-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:46.314610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.314607 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:46.314610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.314620 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/56158aa7-e84a-405f-b077-19bb9055f976-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:46.314879 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.314633 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/56158aa7-e84a-405f-b077-19bb9055f976-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:46.314879 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.314644 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b77gw\" (UniqueName: \"kubernetes.io/projected/56158aa7-e84a-405f-b077-19bb9055f976-kube-api-access-b77gw\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:15:46.582168 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.582068 5902 generic.go:356] "Generic (PLEG): container finished" podID="56158aa7-e84a-405f-b077-19bb9055f976" containerID="d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49" exitCode=0 Oct 30 03:15:46.582168 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.582094 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerDied","Data":"d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49"} Oct 30 03:15:46.582168 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.582139 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" Oct 30 03:15:46.582386 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.582176 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5" event={"ID":"56158aa7-e84a-405f-b077-19bb9055f976","Type":"ContainerDied","Data":"70163823bc679ecc8f806082cc6c22cfa46c2d7070bddeca168bea0d98a40944"} Oct 30 03:15:46.582386 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.582225 5902 scope.go:122] "RemoveContainer" containerID="d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49" Oct 30 03:15:46.590813 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.590793 5902 scope.go:122] "RemoveContainer" containerID="90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593" Oct 30 03:15:46.597928 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.597913 5902 scope.go:122] "RemoveContainer" containerID="07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451" Oct 30 03:15:46.604572 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.604555 5902 scope.go:122] "RemoveContainer" containerID="8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac" Oct 30 03:15:46.614897 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.614875 5902 scope.go:122] "RemoveContainer" containerID="d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49" Oct 30 03:15:46.615159 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:46.615136 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49\": container with ID starting with d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49 not found: ID does not exist" containerID="d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49" Oct 30 03:15:46.615218 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.615167 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49"} err="failed to get container status \"d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49\": rpc error: code = NotFound desc = could not find container \"d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49\": container with ID starting with d14e3ffc202c22026fdfbcde86949367f505c82e61cd3f92670ce7219eda1a49 not found: ID does not exist" Oct 30 03:15:46.615218 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.615186 5902 scope.go:122] "RemoveContainer" containerID="90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593" Oct 30 03:15:46.615468 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:46.615446 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593\": container with ID starting with 90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593 not found: ID does not exist" containerID="90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593" Oct 30 03:15:46.615529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.615474 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593"} err="failed to get container status \"90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593\": rpc error: code = NotFound desc = could not find container \"90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593\": container with ID starting with 90d7611ee2c6d77c0208faf502ea37762dbdcf8a827eef2353186606d33de593 not found: ID does not exist" Oct 30 03:15:46.615529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.615490 5902 scope.go:122] "RemoveContainer" containerID="07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451" Oct 30 03:15:46.615737 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:46.615718 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451\": container with ID starting with 07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451 not found: ID does not exist" containerID="07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451" Oct 30 03:15:46.615794 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.615742 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451"} err="failed to get container status \"07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451\": rpc error: code = NotFound desc = could not find container \"07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451\": container with ID starting with 07304efcff1c6f78136cfc8627e185af2a5584e47691903d7916c10c746f7451 not found: ID does not exist" Oct 30 03:15:46.615794 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.615755 5902 scope.go:122] "RemoveContainer" containerID="8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac" Oct 30 03:15:46.616009 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:15:46.615989 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac\": container with ID starting with 8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac not found: ID does not exist" containerID="8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac" Oct 30 03:15:46.616061 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.616015 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac"} err="failed to get container status \"8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac\": rpc error: code = NotFound desc = could not find container \"8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac\": container with ID starting with 8b83f5c75875cf9dcc44ed41f706b60250b78eccbe903354b9da5a61f49adaac not found: ID does not exist" Oct 30 03:15:46.644315 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.644289 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5"] Oct 30 03:15:46.655415 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:46.655391 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-e6d7890379/isvc-logger-raw-27bd3-predictor-79d667c89c-5lwg5"] Oct 30 03:15:47.018903 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:15:47.018872 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56158aa7-e84a-405f-b077-19bb9055f976" path="/var/lib/kubelet/pods/56158aa7-e84a-405f-b077-19bb9055f976/volumes" Oct 30 03:16:11.719762 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:16:11.719684 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:16:14.596147 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596114 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm"] Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596395 5902 cpu_manager.go:414] "containerMap: removing container" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596405 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596425 5902 cpu_manager.go:414] "containerMap: removing container" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596431 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596440 5902 cpu_manager.go:414] "containerMap: removing container" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kserve-container" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596445 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kserve-container" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596451 5902 cpu_manager.go:414] "containerMap: removing container" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kube-rbac-proxy" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596456 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kube-rbac-proxy" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596490 5902 cpu_manager.go:414] "containerMap: removing container" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596495 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596503 5902 cpu_manager.go:414] "containerMap: removing container" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="storage-initializer" Oct 30 03:16:14.596535 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596509 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="storage-initializer" Oct 30 03:16:14.597001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596553 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="agent" Oct 30 03:16:14.597001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596563 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kube-rbac-proxy" Oct 30 03:16:14.597001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596568 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kserve-container" Oct 30 03:16:14.597001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596574 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="b795fb8b-9080-43de-9df3-9e359d6bfd32" containerName="kube-rbac-proxy" Oct 30 03:16:14.597001 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.596582 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="56158aa7-e84a-405f-b077-19bb9055f976" containerName="kserve-container" Oct 30 03:16:14.599742 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.599720 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.613862 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.613840 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"seaweedfs-s3-creds\"" Oct 30 03:16:14.623309 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.623279 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.623414 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.623336 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.623459 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.623414 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8scj\" (UniqueName: \"kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.623459 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.623440 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.623539 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.623509 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.624366 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.624340 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"raw-sklearn-runtime-91d05-kube-rbac-proxy-sar-config\"" Oct 30 03:16:14.624475 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.624369 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"openshift-service-ca.crt\"" Oct 30 03:16:14.624475 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.624430 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"default-dockercfg-57rzd\"" Oct 30 03:16:14.624475 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.624451 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"raw-sklearn-runtime-91d05-predictor-serving-cert\"" Oct 30 03:16:14.624475 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.624464 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"kube-root-ca.crt\"" Oct 30 03:16:14.624658 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.624591 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:16:14.625097 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.625076 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm"] Oct 30 03:16:14.724057 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724014 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724263 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724077 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724263 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724105 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8scj\" (UniqueName: \"kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724263 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724133 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724263 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724164 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724582 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724561 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724890 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724869 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.724947 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.724869 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.726693 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.726669 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.746523 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.746487 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8scj\" (UniqueName: \"kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj\") pod \"raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:14.915869 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:14.915777 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:15.223916 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:15.223884 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm"] Oct 30 03:16:15.227219 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:16:15.227177 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5c7bf2_c28a_4e92_9eb9_f9a433cac48d.slice/crio-55cb566f3f05fc17d79616351326104fc0f402468b72a479ab226cd3b03ed1d8 WatchSource:0}: Error finding container 55cb566f3f05fc17d79616351326104fc0f402468b72a479ab226cd3b03ed1d8: Status 404 returned error can't find the container with id 55cb566f3f05fc17d79616351326104fc0f402468b72a479ab226cd3b03ed1d8 Oct 30 03:16:15.686930 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:15.686895 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerStarted","Data":"cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c"} Oct 30 03:16:15.687294 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:15.686951 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerStarted","Data":"55cb566f3f05fc17d79616351326104fc0f402468b72a479ab226cd3b03ed1d8"} Oct 30 03:16:19.701765 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:19.701735 5902 generic.go:356] "Generic (PLEG): container finished" podID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerID="cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c" exitCode=0 Oct 30 03:16:19.702201 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:19.701831 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerDied","Data":"cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c"} Oct 30 03:16:20.707405 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:20.707367 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerStarted","Data":"027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089"} Oct 30 03:16:20.707945 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:20.707444 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerStarted","Data":"d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8"} Oct 30 03:16:20.707945 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:20.707674 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:20.761833 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:20.761765 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podStartSLOduration=3.049098265 podStartE2EDuration="6.761753052s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.712654787s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:16:20.760984681 +0000 UTC m=+1193.858401240" watchObservedRunningTime="2026-10-30 03:16:20.761753052 +0000 UTC m=+1193.859169611" Oct 30 03:16:21.711506 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:21.711470 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:21.712880 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:21.712851 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:16:22.715654 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:22.715611 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:16:27.720486 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:27.720458 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:16:27.721030 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:27.721004 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:16:37.721320 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:37.721286 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:16:41.719415 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:16:41.719372 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:16:47.721047 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:47.721012 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:16:57.720928 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:16:57.720890 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:17:07.721553 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:07.721515 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:17:11.719379 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:17:11.719341 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:17:17.721854 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:17.721799 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:17:27.721542 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:27.721505 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:17:37.721873 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:37.721784 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:17:41.719785 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:17:41.719741 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:17:45.389741 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:45.389707 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm"] Oct 30 03:17:45.390152 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:45.389986 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" containerID="cri-o://d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8" gracePeriod=30 Oct 30 03:17:45.390152 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:45.390061 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kube-rbac-proxy" containerID="cri-o://027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089" gracePeriod=30 Oct 30 03:17:46.014404 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:46.014375 5902 generic.go:356] "Generic (PLEG): container finished" podID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerID="027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089" exitCode=0 Oct 30 03:17:46.014581 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:46.014455 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerDied","Data":"027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089"} Oct 30 03:17:47.716205 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:47.716155 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused" Oct 30 03:17:47.721622 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:47.721592 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused" Oct 30 03:17:50.435652 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.435626 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:17:50.518224 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518146 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config\") pod \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " Oct 30 03:17:50.518224 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518190 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls\") pod \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " Oct 30 03:17:50.518450 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518249 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert\") pod \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " Oct 30 03:17:50.518450 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518281 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location\") pod \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " Oct 30 03:17:50.518450 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518301 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj\" (UniqueName: \"kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj\") pod \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\" (UID: \"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d\") " Oct 30 03:17:50.518682 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518654 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config" pod "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" (UID: "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:17:50.518804 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518660 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location" pod "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" (UID: "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:17:50.518804 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.518695 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert" pod "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" (UID: "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:17:50.520458 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.520437 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls" pod "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" (UID: "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:17:50.520521 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.520488 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj" pod "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" (UID: "7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d"). InnerVolumeSpecName "kube-api-access-q8scj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:17:50.619835 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.619793 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:17:50.619835 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.619838 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:17:50.620041 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.619848 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:17:50.620041 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.619856 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:17:50.620041 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:50.619865 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8scj\" (UniqueName: \"kubernetes.io/projected/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d-kube-api-access-q8scj\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:17:51.034596 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.034565 5902 generic.go:356] "Generic (PLEG): container finished" podID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerID="d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8" exitCode=0 Oct 30 03:17:51.034738 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.034613 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerDied","Data":"d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8"} Oct 30 03:17:51.034738 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.034633 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" Oct 30 03:17:51.034738 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.034657 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm" event={"ID":"7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d","Type":"ContainerDied","Data":"55cb566f3f05fc17d79616351326104fc0f402468b72a479ab226cd3b03ed1d8"} Oct 30 03:17:51.034738 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.034682 5902 scope.go:122] "RemoveContainer" containerID="027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089" Oct 30 03:17:51.043447 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.043430 5902 scope.go:122] "RemoveContainer" containerID="d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8" Oct 30 03:17:51.050516 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.050500 5902 scope.go:122] "RemoveContainer" containerID="cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c" Oct 30 03:17:51.057388 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.057368 5902 scope.go:122] "RemoveContainer" containerID="027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089" Oct 30 03:17:51.057667 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:17:51.057647 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089\": container with ID starting with 027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089 not found: ID does not exist" containerID="027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089" Oct 30 03:17:51.057716 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.057676 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089"} err="failed to get container status \"027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089\": rpc error: code = NotFound desc = could not find container \"027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089\": container with ID starting with 027697033f7df035e9049678aeed1cf449bf4ff0675276af4525aed38e916089 not found: ID does not exist" Oct 30 03:17:51.057716 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.057695 5902 scope.go:122] "RemoveContainer" containerID="d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8" Oct 30 03:17:51.057945 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:17:51.057926 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8\": container with ID starting with d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8 not found: ID does not exist" containerID="d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8" Oct 30 03:17:51.058018 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.057955 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8"} err="failed to get container status \"d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8\": rpc error: code = NotFound desc = could not find container \"d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8\": container with ID starting with d0894b3487cdacab2457fe8667739d16483e2fd7e260ea206bb8efb0ccc66fd8 not found: ID does not exist" Oct 30 03:17:51.058018 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.057978 5902 scope.go:122] "RemoveContainer" containerID="cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c" Oct 30 03:17:51.058233 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:17:51.058215 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c\": container with ID starting with cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c not found: ID does not exist" containerID="cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c" Oct 30 03:17:51.058287 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.058237 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c"} err="failed to get container status \"cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c\": rpc error: code = NotFound desc = could not find container \"cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c\": container with ID starting with cb96603bc5d4e810040185c8a1880b07c07feb740476cd1583a0c92f41b75f7c not found: ID does not exist" Oct 30 03:17:51.097435 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.097410 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm"] Oct 30 03:17:51.111122 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:51.111097 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/raw-sklearn-runtime-91d05-predictor-67846c6579-rmlrm"] Oct 30 03:17:53.021273 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.021242 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" path="/var/lib/kubelet/pods/7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d/volumes" Oct 30 03:17:53.066212 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066181 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj"] Oct 30 03:17:53.066494 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066481 5902 cpu_manager.go:414] "containerMap: removing container" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="storage-initializer" Oct 30 03:17:53.066531 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066496 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="storage-initializer" Oct 30 03:17:53.066531 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066518 5902 cpu_manager.go:414] "containerMap: removing container" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" Oct 30 03:17:53.066531 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066523 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" Oct 30 03:17:53.066645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066543 5902 cpu_manager.go:414] "containerMap: removing container" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kube-rbac-proxy" Oct 30 03:17:53.066645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066549 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kube-rbac-proxy" Oct 30 03:17:53.066645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066591 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kube-rbac-proxy" Oct 30 03:17:53.066645 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.066602 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b5c7bf2-c28a-4e92-9eb9-f9a433cac48d" containerName="kserve-container" Oct 30 03:17:53.071308 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.071288 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.096125 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096098 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"kube-root-ca.crt\"" Oct 30 03:17:53.096255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096145 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"openshift-service-ca.crt\"" Oct 30 03:17:53.096255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096144 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:17:53.096255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096145 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\"" Oct 30 03:17:53.096255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096185 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"default-dockercfg-57rzd\"" Oct 30 03:17:53.096255 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096226 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"seaweedfs-s3-creds\"" Oct 30 03:17:53.096486 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.096282 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-create-v2-predictor-serving-cert\"" Oct 30 03:17:53.099232 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.099210 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj"] Oct 30 03:17:53.137542 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.137514 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.137728 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.137554 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.137728 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.137572 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgzvn\" (UniqueName: \"kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.137728 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.137691 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.137728 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.137718 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.238927 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.238891 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.238927 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.238929 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.239181 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.238950 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgzvn\" (UniqueName: \"kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.239181 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.238977 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.239181 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.239007 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.239393 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.239374 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.239603 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.239583 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.239652 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.239633 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.241623 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.241597 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.248962 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.248935 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj"] Oct 30 03:17:53.252661 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.252643 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.264080 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.264053 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgzvn\" (UniqueName: \"kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn\") pod \"isvc-canary-create-v2-predictor-78fcf67857-wx9jj\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.274495 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.274425 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj"] Oct 30 03:17:53.287057 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.287030 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-create-predictor-serving-cert\"" Oct 30 03:17:53.339890 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.339856 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.340054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.339903 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.340054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.339969 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7fs\" (UniqueName: \"kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.340124 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.340073 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.340124 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.340094 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.388178 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.388152 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:53.441282 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.441250 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7fs\" (UniqueName: \"kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.441417 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.441303 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.441417 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.441318 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.441417 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.441360 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.441417 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.441400 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.442282 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.442222 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.442282 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.442222 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.442503 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.442446 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.444528 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.444509 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.465298 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.465275 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7fs\" (UniqueName: \"kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs\") pod \"isvc-canary-create-predictor-57bc666c8b-9gzgj\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.569184 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.569102 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:53.698955 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.698925 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj"] Oct 30 03:17:53.702066 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:17:53.702040 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadfb850d_5a34_444c_876d_f12205c96e18.slice/crio-f066b9d5f5bedc4ea9922e464735c0d442c163d18891b2e6f7bdd92e61a9ebc4 WatchSource:0}: Error finding container f066b9d5f5bedc4ea9922e464735c0d442c163d18891b2e6f7bdd92e61a9ebc4: Status 404 returned error can't find the container with id f066b9d5f5bedc4ea9922e464735c0d442c163d18891b2e6f7bdd92e61a9ebc4 Oct 30 03:17:53.867408 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:53.867332 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj"] Oct 30 03:17:53.870717 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:17:53.870692 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2bf4ef_61e8_4ed3_8b3f_b677ce06ca55.slice/crio-7ded7a0f94fb1588e6d8fa57cb10d22425f210592321633da4d41e3fc30fb55d WatchSource:0}: Error finding container 7ded7a0f94fb1588e6d8fa57cb10d22425f210592321633da4d41e3fc30fb55d: Status 404 returned error can't find the container with id 7ded7a0f94fb1588e6d8fa57cb10d22425f210592321633da4d41e3fc30fb55d Oct 30 03:17:54.047347 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:54.047309 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerStarted","Data":"97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634"} Oct 30 03:17:54.047924 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:54.047379 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerStarted","Data":"f066b9d5f5bedc4ea9922e464735c0d442c163d18891b2e6f7bdd92e61a9ebc4"} Oct 30 03:17:54.049041 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:54.049005 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerStarted","Data":"06e12581ff526c855a99c67de899a6daa63583f2713f7db6f7fc94a3c8543c09"} Oct 30 03:17:54.049176 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:54.049051 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerStarted","Data":"7ded7a0f94fb1588e6d8fa57cb10d22425f210592321633da4d41e3fc30fb55d"} Oct 30 03:17:58.070551 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:58.070514 5902 generic.go:356] "Generic (PLEG): container finished" podID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerID="06e12581ff526c855a99c67de899a6daa63583f2713f7db6f7fc94a3c8543c09" exitCode=0 Oct 30 03:17:58.071026 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:58.070586 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerDied","Data":"06e12581ff526c855a99c67de899a6daa63583f2713f7db6f7fc94a3c8543c09"} Oct 30 03:17:58.072192 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:58.072171 5902 generic.go:356] "Generic (PLEG): container finished" podID="adfb850d-5a34-444c-876d-f12205c96e18" containerID="97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634" exitCode=0 Oct 30 03:17:58.072304 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:58.072236 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerDied","Data":"97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634"} Oct 30 03:17:59.078671 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.078634 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerStarted","Data":"9973c3686207323dd288776f81af6724f1c76b59bdb639d0a3e5630d6375a61b"} Oct 30 03:17:59.079172 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.078702 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerStarted","Data":"84d677c83fad619d86fb4bedfaa34838f7cd38736fae5cb54f819521429ca01b"} Oct 30 03:17:59.079172 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.078894 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:17:59.080789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.080767 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerStarted","Data":"e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf"} Oct 30 03:17:59.080911 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.080812 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerStarted","Data":"456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e"} Oct 30 03:17:59.081062 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.081049 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:59.081098 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.081088 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:17:59.082321 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.082297 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:17:59.133769 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.133704 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podStartSLOduration=2.338490745 podStartE2EDuration="6.133688854s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.795198109s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:59.13311203 +0000 UTC m=+1292.230528591" watchObservedRunningTime="2026-10-30 03:17:59.133688854 +0000 UTC m=+1292.231105417" Oct 30 03:17:59.177641 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:17:59.177588 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podStartSLOduration=2.278213325 podStartE2EDuration="6.177571794s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.899358469s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:17:59.17719286 +0000 UTC m=+1292.274609419" watchObservedRunningTime="2026-10-30 03:17:59.177571794 +0000 UTC m=+1292.274988353" Oct 30 03:18:00.085112 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:00.085073 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:00.085483 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:00.085158 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:18:00.086383 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:00.086360 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:01.088622 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:01.088580 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:05.089329 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:05.089296 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:18:05.089980 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:05.089952 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:06.093412 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:06.093380 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:18:06.094020 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:06.093992 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:11.718788 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:18:11.718748 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:18:15.090247 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:15.090207 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:16.093992 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:16.093954 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:25.090268 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:25.090226 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:26.094981 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:26.094940 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:35.090330 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:35.090291 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:36.094789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:36.094753 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:41.719366 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:18:41.719332 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:18:45.090008 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:45.089966 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:46.094561 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:46.094518 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:18:55.089933 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:55.089889 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:18:56.094888 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:18:56.094846 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:19:05.090800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:05.090762 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:19:06.094801 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:06.094763 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:19:08.014024 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:08.013937 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:19:11.719736 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:19:11.719688 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:19:15.090263 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:15.090209 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:19:18.013973 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:18.013933 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused" Oct 30 03:19:25.090774 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:25.090745 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:19:25.562052 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:25.562022 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj"] Oct 30 03:19:25.562314 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:25.562275 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" containerID="cri-o://84d677c83fad619d86fb4bedfaa34838f7cd38736fae5cb54f819521429ca01b" gracePeriod=30 Oct 30 03:19:25.562404 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:25.562315 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kube-rbac-proxy" containerID="cri-o://9973c3686207323dd288776f81af6724f1c76b59bdb639d0a3e5630d6375a61b" gracePeriod=30 Oct 30 03:19:25.563165 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:25.562988 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:19:26.093959 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:26.089028 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused" Oct 30 03:19:26.413202 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:26.413125 5902 generic.go:356] "Generic (PLEG): container finished" podID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerID="9973c3686207323dd288776f81af6724f1c76b59bdb639d0a3e5630d6375a61b" exitCode=0 Oct 30 03:19:26.413202 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:26.413166 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerDied","Data":"9973c3686207323dd288776f81af6724f1c76b59bdb639d0a3e5630d6375a61b"} Oct 30 03:19:30.431412 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.431376 5902 generic.go:356] "Generic (PLEG): container finished" podID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerID="84d677c83fad619d86fb4bedfaa34838f7cd38736fae5cb54f819521429ca01b" exitCode=0 Oct 30 03:19:30.431776 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.431437 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerDied","Data":"84d677c83fad619d86fb4bedfaa34838f7cd38736fae5cb54f819521429ca01b"} Oct 30 03:19:30.507286 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.507262 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:19:30.569395 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569364 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location\") pod \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " Oct 30 03:19:30.569576 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569413 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs\" (UniqueName: \"kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs\") pod \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " Oct 30 03:19:30.569576 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569436 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config\") pod \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " Oct 30 03:19:30.569576 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569454 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls\") pod \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " Oct 30 03:19:30.569576 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569528 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert\") pod \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\" (UID: \"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55\") " Oct 30 03:19:30.569879 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569835 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location" pod "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" (UID: "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:19:30.569960 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569876 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config" pod "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" (UID: "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:19:30.569960 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.569908 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert" pod "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" (UID: "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:19:30.571691 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.571661 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs" pod "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" (UID: "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55"). InnerVolumeSpecName "kube-api-access-gn7fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:19:30.571799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.571758 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls" pod "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" (UID: "5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:19:30.671221 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.671134 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:30.671221 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.671168 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:30.671221 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.671178 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn7fs\" (UniqueName: \"kubernetes.io/projected/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-api-access-gn7fs\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:30.671221 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.671187 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:30.671221 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:30.671196 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:31.437399 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.437367 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" Oct 30 03:19:31.437399 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.437378 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj" event={"ID":"5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55","Type":"ContainerDied","Data":"7ded7a0f94fb1588e6d8fa57cb10d22425f210592321633da4d41e3fc30fb55d"} Oct 30 03:19:31.437919 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.437448 5902 scope.go:122] "RemoveContainer" containerID="9973c3686207323dd288776f81af6724f1c76b59bdb639d0a3e5630d6375a61b" Oct 30 03:19:31.445633 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.445612 5902 scope.go:122] "RemoveContainer" containerID="84d677c83fad619d86fb4bedfaa34838f7cd38736fae5cb54f819521429ca01b" Oct 30 03:19:31.452902 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.452879 5902 scope.go:122] "RemoveContainer" containerID="06e12581ff526c855a99c67de899a6daa63583f2713f7db6f7fc94a3c8543c09" Oct 30 03:19:31.496699 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.496660 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj"] Oct 30 03:19:31.507220 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:31.507192 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-predictor-57bc666c8b-9gzgj"] Oct 30 03:19:33.018827 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:33.018789 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" path="/var/lib/kubelet/pods/5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55/volumes" Oct 30 03:19:33.144400 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:33.144367 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj"] Oct 30 03:19:33.144668 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:33.144623 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" containerID="cri-o://456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e" gracePeriod=30 Oct 30 03:19:33.144899 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:33.144721 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kube-rbac-proxy" containerID="cri-o://e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf" gracePeriod=30 Oct 30 03:19:33.447179 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:33.447145 5902 generic.go:356] "Generic (PLEG): container finished" podID="adfb850d-5a34-444c-876d-f12205c96e18" containerID="e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf" exitCode=0 Oct 30 03:19:33.447325 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:33.447213 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerDied","Data":"e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf"} Oct 30 03:19:35.086171 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:35.086131 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused" Oct 30 03:19:35.090517 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:35.090491 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused" Oct 30 03:19:37.900575 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.900551 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:19:37.935413 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935384 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert\") pod \"adfb850d-5a34-444c-876d-f12205c96e18\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " Oct 30 03:19:37.935568 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935421 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location\") pod \"adfb850d-5a34-444c-876d-f12205c96e18\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " Oct 30 03:19:37.935568 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935463 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls\") pod \"adfb850d-5a34-444c-876d-f12205c96e18\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " Oct 30 03:19:37.935661 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935575 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config\") pod \"adfb850d-5a34-444c-876d-f12205c96e18\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " Oct 30 03:19:37.935661 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935650 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn\" (UniqueName: \"kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn\") pod \"adfb850d-5a34-444c-876d-f12205c96e18\" (UID: \"adfb850d-5a34-444c-876d-f12205c96e18\") " Oct 30 03:19:37.935831 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935793 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location" pod "adfb850d-5a34-444c-876d-f12205c96e18" (UID: "adfb850d-5a34-444c-876d-f12205c96e18"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:19:37.935910 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935806 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert" pod "adfb850d-5a34-444c-876d-f12205c96e18" (UID: "adfb850d-5a34-444c-876d-f12205c96e18"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:19:37.935910 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935885 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config" pod "adfb850d-5a34-444c-876d-f12205c96e18" (UID: "adfb850d-5a34-444c-876d-f12205c96e18"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:19:37.936019 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935917 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:37.936019 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.935931 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/adfb850d-5a34-444c-876d-f12205c96e18-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:37.937547 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.937528 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls" pod "adfb850d-5a34-444c-876d-f12205c96e18" (UID: "adfb850d-5a34-444c-876d-f12205c96e18"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:19:37.937665 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:37.937645 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn" pod "adfb850d-5a34-444c-876d-f12205c96e18" (UID: "adfb850d-5a34-444c-876d-f12205c96e18"). InnerVolumeSpecName "kube-api-access-hgzvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:19:38.036598 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.036511 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/adfb850d-5a34-444c-876d-f12205c96e18-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:38.036598 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.036542 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/adfb850d-5a34-444c-876d-f12205c96e18-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:38.036598 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.036556 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgzvn\" (UniqueName: \"kubernetes.io/projected/adfb850d-5a34-444c-876d-f12205c96e18-kube-api-access-hgzvn\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:19:38.468246 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.468215 5902 generic.go:356] "Generic (PLEG): container finished" podID="adfb850d-5a34-444c-876d-f12205c96e18" containerID="456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e" exitCode=0 Oct 30 03:19:38.468413 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.468273 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" Oct 30 03:19:38.468413 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.468290 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerDied","Data":"456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e"} Oct 30 03:19:38.468413 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.468344 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj" event={"ID":"adfb850d-5a34-444c-876d-f12205c96e18","Type":"ContainerDied","Data":"f066b9d5f5bedc4ea9922e464735c0d442c163d18891b2e6f7bdd92e61a9ebc4"} Oct 30 03:19:38.468413 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.468372 5902 scope.go:122] "RemoveContainer" containerID="e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf" Oct 30 03:19:38.476570 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.476428 5902 scope.go:122] "RemoveContainer" containerID="456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e" Oct 30 03:19:38.483750 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.483726 5902 scope.go:122] "RemoveContainer" containerID="97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634" Oct 30 03:19:38.490789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.490765 5902 scope.go:122] "RemoveContainer" containerID="e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf" Oct 30 03:19:38.491141 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:19:38.491120 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf\": container with ID starting with e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf not found: ID does not exist" containerID="e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf" Oct 30 03:19:38.491215 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.491148 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf"} err="failed to get container status \"e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf\": rpc error: code = NotFound desc = could not find container \"e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf\": container with ID starting with e53b335d016b34199dddb8815b7797ce29d2be0e4cc5cb1b0ea4736493a7b0bf not found: ID does not exist" Oct 30 03:19:38.491215 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.491165 5902 scope.go:122] "RemoveContainer" containerID="456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e" Oct 30 03:19:38.491401 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:19:38.491386 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e\": container with ID starting with 456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e not found: ID does not exist" containerID="456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e" Oct 30 03:19:38.491441 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.491404 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e"} err="failed to get container status \"456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e\": rpc error: code = NotFound desc = could not find container \"456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e\": container with ID starting with 456d4d560dcd9a17ed297ace946dc66f9e1b31c4ffc33363259de4fa4812612e not found: ID does not exist" Oct 30 03:19:38.491441 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.491418 5902 scope.go:122] "RemoveContainer" containerID="97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634" Oct 30 03:19:38.491657 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:19:38.491640 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634\": container with ID starting with 97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634 not found: ID does not exist" containerID="97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634" Oct 30 03:19:38.491700 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.491663 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634"} err="failed to get container status \"97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634\": rpc error: code = NotFound desc = could not find container \"97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634\": container with ID starting with 97337c38841d563a318dafc3d511739217996f79b0951c366674e0a43f8de634 not found: ID does not exist" Oct 30 03:19:38.530467 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.530433 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj"] Oct 30 03:19:38.535307 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:38.535285 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-create-v2-predictor-78fcf67857-wx9jj"] Oct 30 03:19:39.018493 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:39.018450 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adfb850d-5a34-444c-876d-f12205c96e18" path="/var/lib/kubelet/pods/adfb850d-5a34-444c-876d-f12205c96e18/volumes" Oct 30 03:19:41.097411 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097380 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw"] Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097646 5902 cpu_manager.go:414] "containerMap: removing container" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="storage-initializer" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097655 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="storage-initializer" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097668 5902 cpu_manager.go:414] "containerMap: removing container" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097673 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097687 5902 cpu_manager.go:414] "containerMap: removing container" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kube-rbac-proxy" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097692 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kube-rbac-proxy" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097701 5902 cpu_manager.go:414] "containerMap: removing container" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="storage-initializer" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097706 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="storage-initializer" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097714 5902 cpu_manager.go:414] "containerMap: removing container" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097721 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097746 5902 cpu_manager.go:414] "containerMap: removing container" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kube-rbac-proxy" Oct 30 03:19:41.097799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097753 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kube-rbac-proxy" Oct 30 03:19:41.098277 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097810 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kserve-container" Oct 30 03:19:41.098277 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097833 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kube-rbac-proxy" Oct 30 03:19:41.098277 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097847 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="adfb850d-5a34-444c-876d-f12205c96e18" containerName="kserve-container" Oct 30 03:19:41.098277 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.097856 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2bf4ef-61e8-4ed3-8b3f-b677ce06ca55" containerName="kube-rbac-proxy" Oct 30 03:19:41.102521 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.102504 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.117740 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.117719 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"default-dockercfg-57rzd\"" Oct 30 03:19:41.117898 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.117719 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"seaweedfs-s3-creds\"" Oct 30 03:19:41.117898 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.117764 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-promote-v2-predictor-serving-cert\"" Oct 30 03:19:41.117898 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.117764 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\"" Oct 30 03:19:41.128610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.128419 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:19:41.128610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.128452 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"openshift-service-ca.crt\"" Oct 30 03:19:41.128610 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.128453 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw1-d44ea3c803\"/\"kube-root-ca.crt\"" Oct 30 03:19:41.129269 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.129251 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw"] Oct 30 03:19:41.163316 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.163287 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.163415 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.163327 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.163415 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.163350 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.163486 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.163454 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.163486 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.163478 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99gc6\" (UniqueName: \"kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.264590 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.264552 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99gc6\" (UniqueName: \"kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.264768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.264641 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.264768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.264681 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.264768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.264705 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.264768 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.264730 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.265250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.265227 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.265562 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.265542 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.265562 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.265553 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.267133 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.267114 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.293785 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.293766 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99gc6\" (UniqueName: \"kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6\") pod \"isvc-canary-promote-v2-predictor-788cb55695-hsdgw\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.319788 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.319760 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8"] Oct 30 03:19:41.323497 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.323479 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.338640 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.338621 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw1-d44ea3c803\"/\"isvc-canary-promote-predictor-serving-cert\"" Oct 30 03:19:41.345950 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.345929 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8"] Oct 30 03:19:41.365754 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.365694 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kh2l\" (UniqueName: \"kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.365754 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.365737 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.366068 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.365765 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.366068 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.365976 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.366068 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.366041 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.418748 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.418717 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.467217 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.467263 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kh2l\" (UniqueName: \"kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.467306 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.467333 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.467396 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.468063 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.468654 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.468800 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.468764 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.474400 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.474358 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.490885 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.490835 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kh2l\" (UniqueName: \"kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l\") pod \"isvc-canary-promote-predictor-57ffc5b69d-sdsh8\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.639916 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.639799 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:41.722185 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:19:41.722131 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:19:41.729193 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.729103 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw"] Oct 30 03:19:41.731865 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:19:41.731802 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8d3bb7_3a69_46ee_a106_7704dac39099.slice/crio-c7ed0173f0cedd16f17ea2a613c8e621e3526629ac7540800391a7b5889fe5a8 WatchSource:0}: Error finding container c7ed0173f0cedd16f17ea2a613c8e621e3526629ac7540800391a7b5889fe5a8: Status 404 returned error can't find the container with id c7ed0173f0cedd16f17ea2a613c8e621e3526629ac7540800391a7b5889fe5a8 Oct 30 03:19:41.937808 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:41.937777 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8"] Oct 30 03:19:41.940683 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:19:41.940655 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c802f2_0c1a_41ae_a326_cd55dbf04a31.slice/crio-da7ea80dcc721ee9b3248b091c2284818aef7e1cd0ab347528757a184b47f052 WatchSource:0}: Error finding container da7ea80dcc721ee9b3248b091c2284818aef7e1cd0ab347528757a184b47f052: Status 404 returned error can't find the container with id da7ea80dcc721ee9b3248b091c2284818aef7e1cd0ab347528757a184b47f052 Oct 30 03:19:42.486304 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:42.486267 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerStarted","Data":"bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1"} Oct 30 03:19:42.486691 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:42.486342 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerStarted","Data":"da7ea80dcc721ee9b3248b091c2284818aef7e1cd0ab347528757a184b47f052"} Oct 30 03:19:42.488332 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:42.488307 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerStarted","Data":"df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b"} Oct 30 03:19:42.488466 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:42.488356 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerStarted","Data":"c7ed0173f0cedd16f17ea2a613c8e621e3526629ac7540800391a7b5889fe5a8"} Oct 30 03:19:46.505706 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:46.505667 5902 generic.go:356] "Generic (PLEG): container finished" podID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerID="df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b" exitCode=0 Oct 30 03:19:46.506214 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:46.505716 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerDied","Data":"df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b"} Oct 30 03:19:46.507515 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:46.507483 5902 generic.go:356] "Generic (PLEG): container finished" podID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerID="bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1" exitCode=0 Oct 30 03:19:46.507562 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:46.507531 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerDied","Data":"bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1"} Oct 30 03:19:47.513091 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.513056 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerStarted","Data":"8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e"} Oct 30 03:19:47.513566 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.513123 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerStarted","Data":"07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa"} Oct 30 03:19:47.513566 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.513330 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:47.515206 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.515184 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerStarted","Data":"74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22"} Oct 30 03:19:47.515333 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.515230 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerStarted","Data":"7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91"} Oct 30 03:19:47.515432 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.515410 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:47.566954 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.566903 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podStartSLOduration=2.8344377769999998 podStartE2EDuration="6.56688959s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.732451813s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:47.566332597 +0000 UTC m=+1400.663749193" watchObservedRunningTime="2026-10-30 03:19:47.56688959 +0000 UTC m=+1400.664306148" Oct 30 03:19:47.611809 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:47.611736 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podStartSLOduration=2.634052735 podStartE2EDuration="6.611721901s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.977669166s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:19:47.611004958 +0000 UTC m=+1400.708421511" watchObservedRunningTime="2026-10-30 03:19:47.611721901 +0000 UTC m=+1400.709138453" Oct 30 03:19:48.519749 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:48.519709 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:48.520206 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:48.519811 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:48.520772 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:48.520737 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:19:48.520907 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:48.520746 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:19:49.524160 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:49.524112 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:19:49.524644 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:49.524213 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:19:54.529359 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:54.529324 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:19:54.529886 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:54.529866 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:19:54.529974 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:54.529856 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:19:54.530332 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:19:54.530308 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:04.530391 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:04.530352 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:20:04.530893 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:04.530352 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:11.718755 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:20:11.718715 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:20:14.530515 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:14.530476 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:20:14.530914 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:14.530475 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:24.529962 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:24.529924 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:20:24.530530 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:24.530437 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:34.530929 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:34.530889 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:34.531447 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:34.530965 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:20:41.720573 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:20:41.720452 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:20:44.529881 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:44.529842 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:20:44.530442 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:44.530413 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:54.530188 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:54.530141 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:20:54.530699 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:54.530276 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:20:57.015704 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:20:57.015663 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused" Oct 30 03:21:04.530534 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:04.530480 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:21:07.017932 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:07.017905 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:21:11.718554 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:21:11.718517 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:21:14.530745 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:14.530713 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:21:21.217132 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:21.217100 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8"] Oct 30 03:21:21.217621 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:21.217365 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" containerID="cri-o://07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa" gracePeriod=30 Oct 30 03:21:21.217621 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:21.217433 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kube-rbac-proxy" containerID="cri-o://8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e" gracePeriod=30 Oct 30 03:21:21.867314 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:21.867281 5902 generic.go:356] "Generic (PLEG): container finished" podID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerID="8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e" exitCode=0 Oct 30 03:21:21.867492 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:21.867352 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerDied","Data":"8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e"} Oct 30 03:21:24.524479 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:24.524431 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused" Oct 30 03:21:24.531006 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:24.530966 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused" Oct 30 03:21:26.166420 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.166396 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:21:26.265035 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265000 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l\" (UniqueName: \"kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l\") pod \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " Oct 30 03:21:26.265213 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265065 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config\") pod \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " Oct 30 03:21:26.265213 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265091 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location\") pod \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " Oct 30 03:21:26.265213 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265134 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert\") pod \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " Oct 30 03:21:26.265213 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265176 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls\" (UniqueName: \"kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls\") pod \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\" (UID: \"94c802f2-0c1a-41ae-a326-cd55dbf04a31\") " Oct 30 03:21:26.265487 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265457 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location" pod "94c802f2-0c1a-41ae-a326-cd55dbf04a31" (UID: "94c802f2-0c1a-41ae-a326-cd55dbf04a31"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:21:26.265560 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265503 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config" pod "94c802f2-0c1a-41ae-a326-cd55dbf04a31" (UID: "94c802f2-0c1a-41ae-a326-cd55dbf04a31"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:26.265560 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.265512 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert" pod "94c802f2-0c1a-41ae-a326-cd55dbf04a31" (UID: "94c802f2-0c1a-41ae-a326-cd55dbf04a31"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:21:26.267203 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.267180 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls" pod "94c802f2-0c1a-41ae-a326-cd55dbf04a31" (UID: "94c802f2-0c1a-41ae-a326-cd55dbf04a31"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:21:26.267310 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.267227 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l" pod "94c802f2-0c1a-41ae-a326-cd55dbf04a31" (UID: "94c802f2-0c1a-41ae-a326-cd55dbf04a31"). InnerVolumeSpecName "kube-api-access-6kh2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:21:26.366047 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.366012 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.366047 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.366039 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.366047 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.366048 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94c802f2-0c1a-41ae-a326-cd55dbf04a31-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.366279 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.366059 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94c802f2-0c1a-41ae-a326-cd55dbf04a31-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.366279 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.366067 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kh2l\" (UniqueName: \"kubernetes.io/projected/94c802f2-0c1a-41ae-a326-cd55dbf04a31-kube-api-access-6kh2l\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:21:26.886533 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.886502 5902 generic.go:356] "Generic (PLEG): container finished" podID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerID="07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa" exitCode=0 Oct 30 03:21:26.886711 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.886559 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" Oct 30 03:21:26.886711 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.886584 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerDied","Data":"07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa"} Oct 30 03:21:26.886711 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.886640 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8" event={"ID":"94c802f2-0c1a-41ae-a326-cd55dbf04a31","Type":"ContainerDied","Data":"da7ea80dcc721ee9b3248b091c2284818aef7e1cd0ab347528757a184b47f052"} Oct 30 03:21:26.886711 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.886668 5902 scope.go:122] "RemoveContainer" containerID="8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e" Oct 30 03:21:26.895169 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.895153 5902 scope.go:122] "RemoveContainer" containerID="07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa" Oct 30 03:21:26.902205 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.902187 5902 scope.go:122] "RemoveContainer" containerID="bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1" Oct 30 03:21:26.909238 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.909224 5902 scope.go:122] "RemoveContainer" containerID="8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e" Oct 30 03:21:26.909481 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:21:26.909465 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e\": container with ID starting with 8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e not found: ID does not exist" containerID="8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e" Oct 30 03:21:26.909521 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.909491 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e"} err="failed to get container status \"8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e\": rpc error: code = NotFound desc = could not find container \"8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e\": container with ID starting with 8653da5ccdd65ec00c4edf68a0fa597faa955d40c94373e23469c1443669ca5e not found: ID does not exist" Oct 30 03:21:26.909521 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.909507 5902 scope.go:122] "RemoveContainer" containerID="07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa" Oct 30 03:21:26.909746 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:21:26.909717 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa\": container with ID starting with 07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa not found: ID does not exist" containerID="07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa" Oct 30 03:21:26.909789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.909753 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa"} err="failed to get container status \"07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa\": rpc error: code = NotFound desc = could not find container \"07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa\": container with ID starting with 07450f5a40e6b5fb87a915895f715d03f26329ae3750b9708e51dd57f48110aa not found: ID does not exist" Oct 30 03:21:26.909789 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.909772 5902 scope.go:122] "RemoveContainer" containerID="bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1" Oct 30 03:21:26.910055 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:21:26.910036 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1\": container with ID starting with bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1 not found: ID does not exist" containerID="bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1" Oct 30 03:21:26.910113 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.910066 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1"} err="failed to get container status \"bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1\": rpc error: code = NotFound desc = could not find container \"bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1\": container with ID starting with bb398e8f4b57b9d2c1c9a1f943ad92c533d02a3aefc0418d3cffffb8d04820d1 not found: ID does not exist" Oct 30 03:21:26.944612 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.944589 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8"] Oct 30 03:21:26.955448 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:26.955426 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-predictor-57ffc5b69d-sdsh8"] Oct 30 03:21:27.019361 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:21:27.019331 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" path="/var/lib/kubelet/pods/94c802f2-0c1a-41ae-a326-cd55dbf04a31/volumes" Oct 30 03:21:41.721871 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:21:41.721832 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:22:11.718847 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:22:11.718754 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:22:41.719862 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:22:41.719804 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:22:51.845485 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:51.845454 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw"] Oct 30 03:22:51.846041 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:51.845694 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" containerID="cri-o://7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91" gracePeriod=30 Oct 30 03:22:51.846041 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:51.845841 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kube-rbac-proxy" containerID="cri-o://74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22" gracePeriod=30 Oct 30 03:22:52.194363 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:52.194331 5902 generic.go:356] "Generic (PLEG): container finished" podID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerID="74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22" exitCode=0 Oct 30 03:22:52.194525 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:52.194405 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerDied","Data":"74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22"} Oct 30 03:22:54.525270 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:54.525229 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused" Oct 30 03:22:56.375251 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.375225 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:22:56.526099 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.525993 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6\" (UniqueName: \"kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6\") pod \"ce8d3bb7-3a69-46ee-a106-7704dac39099\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " Oct 30 03:22:56.526099 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526059 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config\") pod \"ce8d3bb7-3a69-46ee-a106-7704dac39099\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " Oct 30 03:22:56.526099 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526092 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls\") pod \"ce8d3bb7-3a69-46ee-a106-7704dac39099\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " Oct 30 03:22:56.526424 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526127 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location\") pod \"ce8d3bb7-3a69-46ee-a106-7704dac39099\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " Oct 30 03:22:56.526424 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526146 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert\") pod \"ce8d3bb7-3a69-46ee-a106-7704dac39099\" (UID: \"ce8d3bb7-3a69-46ee-a106-7704dac39099\") " Oct 30 03:22:56.526531 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526478 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location" pod "ce8d3bb7-3a69-46ee-a106-7704dac39099" (UID: "ce8d3bb7-3a69-46ee-a106-7704dac39099"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:22:56.526531 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526507 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config" pod "ce8d3bb7-3a69-46ee-a106-7704dac39099" (UID: "ce8d3bb7-3a69-46ee-a106-7704dac39099"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:56.526601 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.526568 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert" pod "ce8d3bb7-3a69-46ee-a106-7704dac39099" (UID: "ce8d3bb7-3a69-46ee-a106-7704dac39099"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:22:56.528215 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.528174 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6" pod "ce8d3bb7-3a69-46ee-a106-7704dac39099" (UID: "ce8d3bb7-3a69-46ee-a106-7704dac39099"). InnerVolumeSpecName "kube-api-access-99gc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:22:56.528323 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.528264 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls" pod "ce8d3bb7-3a69-46ee-a106-7704dac39099" (UID: "ce8d3bb7-3a69-46ee-a106-7704dac39099"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:22:56.627156 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.627119 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce8d3bb7-3a69-46ee-a106-7704dac39099-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:22:56.627156 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.627149 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:22:56.627156 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.627159 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99gc6\" (UniqueName: \"kubernetes.io/projected/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-api-access-99gc6\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:22:56.627390 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.627169 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce8d3bb7-3a69-46ee-a106-7704dac39099-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:22:56.627390 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:56.627178 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce8d3bb7-3a69-46ee-a106-7704dac39099-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:22:57.214787 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.214750 5902 generic.go:356] "Generic (PLEG): container finished" podID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerID="7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91" exitCode=0 Oct 30 03:22:57.215054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.214803 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerDied","Data":"7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91"} Oct 30 03:22:57.215054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.214833 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" Oct 30 03:22:57.215054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.214869 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw" event={"ID":"ce8d3bb7-3a69-46ee-a106-7704dac39099","Type":"ContainerDied","Data":"c7ed0173f0cedd16f17ea2a613c8e621e3526629ac7540800391a7b5889fe5a8"} Oct 30 03:22:57.215054 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.214898 5902 scope.go:122] "RemoveContainer" containerID="74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22" Oct 30 03:22:57.222849 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.222813 5902 scope.go:122] "RemoveContainer" containerID="7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91" Oct 30 03:22:57.229682 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.229663 5902 scope.go:122] "RemoveContainer" containerID="df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b" Oct 30 03:22:57.236547 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.236528 5902 scope.go:122] "RemoveContainer" containerID="74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22" Oct 30 03:22:57.236789 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:22:57.236768 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22\": container with ID starting with 74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22 not found: ID does not exist" containerID="74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22" Oct 30 03:22:57.236908 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.236797 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22"} err="failed to get container status \"74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22\": rpc error: code = NotFound desc = could not find container \"74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22\": container with ID starting with 74e1b469e550429149fbea059c8d2fff9f166b8f299f805b5a700fdc894dfa22 not found: ID does not exist" Oct 30 03:22:57.236908 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.236830 5902 scope.go:122] "RemoveContainer" containerID="7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91" Oct 30 03:22:57.237075 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:22:57.237056 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91\": container with ID starting with 7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91 not found: ID does not exist" containerID="7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91" Oct 30 03:22:57.237115 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.237081 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91"} err="failed to get container status \"7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91\": rpc error: code = NotFound desc = could not find container \"7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91\": container with ID starting with 7741710962d8a0b090bd697eff6313bb6c850f4c009e7f597365bfbe9db26a91 not found: ID does not exist" Oct 30 03:22:57.237115 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.237097 5902 scope.go:122] "RemoveContainer" containerID="df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b" Oct 30 03:22:57.237331 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:22:57.237317 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b\": container with ID starting with df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b not found: ID does not exist" containerID="df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b" Oct 30 03:22:57.237369 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.237337 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b"} err="failed to get container status \"df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b\": rpc error: code = NotFound desc = could not find container \"df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b\": container with ID starting with df0f640e2c424c16cd76c03654b85d17940c8c8af1acb6f5a5ffe18858d1d66b not found: ID does not exist" Oct 30 03:22:57.273622 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.273592 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw"] Oct 30 03:22:57.284210 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:57.284137 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw1-d44ea3c803/isvc-canary-promote-v2-predictor-788cb55695-hsdgw"] Oct 30 03:22:59.018389 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:22:59.018356 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" path="/var/lib/kubelet/pods/ce8d3bb7-3a69-46ee-a106-7704dac39099/volumes" Oct 30 03:23:11.719540 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:23:11.719502 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:23:14.307537 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.307503 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff"] Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.307957 5902 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kube-rbac-proxy" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.307977 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kube-rbac-proxy" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.307995 5902 cpu_manager.go:414] "containerMap: removing container" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308002 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308027 5902 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308034 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308045 5902 cpu_manager.go:414] "containerMap: removing container" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="storage-initializer" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308052 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="storage-initializer" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308065 5902 cpu_manager.go:414] "containerMap: removing container" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kube-rbac-proxy" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308071 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kube-rbac-proxy" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308090 5902 cpu_manager.go:414] "containerMap: removing container" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="storage-initializer" Oct 30 03:23:14.308094 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308098 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="storage-initializer" Oct 30 03:23:14.308702 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308174 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kserve-container" Oct 30 03:23:14.308702 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308188 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kube-rbac-proxy" Oct 30 03:23:14.308702 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308202 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8d3bb7-3a69-46ee-a106-7704dac39099" containerName="kube-rbac-proxy" Oct 30 03:23:14.308702 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.308212 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c802f2-0c1a-41ae-a326-cd55dbf04a31" containerName="kserve-container" Oct 30 03:23:14.312864 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.312843 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.339766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.339734 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-82c29ebf98\"/\"default-dockercfg-t45zf\"" Oct 30 03:23:14.339766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.339753 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-82c29ebf98\"/\"kube-root-ca.crt\"" Oct 30 03:23:14.339766 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.339760 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-82c29ebf98\"/\"openshift-service-ca.crt\"" Oct 30 03:23:14.340014 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.339836 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-82c29ebf98\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\"" Oct 30 03:23:14.340014 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.339849 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-82c29ebf98\"/\"message-dumper-raw-cipn-predictor-serving-cert\"" Oct 30 03:23:14.340979 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.340956 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff"] Oct 30 03:23:14.379519 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.379484 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmz6\" (UniqueName: \"kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.379657 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.379532 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.379657 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.379644 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.480670 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.480629 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmz6\" (UniqueName: \"kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.480880 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.480695 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.480880 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.480748 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.481350 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.481331 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.483189 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.483171 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.503945 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.503921 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmz6\" (UniqueName: \"kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6\") pod \"message-dumper-raw-cipn-predictor-7ff5658b48-s95ff\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.627184 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.627101 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:14.922333 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:14.922246 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff"] Oct 30 03:23:14.925487 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:23:14.925447 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2c0cf6f_bf94_47bb_8df7_7b2b6563f965.slice/crio-91e7da2af1ddb22c37eeeb7ae9b18dbc0a348fc87d41e47382daa2c8dd33b636 WatchSource:0}: Error finding container 91e7da2af1ddb22c37eeeb7ae9b18dbc0a348fc87d41e47382daa2c8dd33b636: Status 404 returned error can't find the container with id 91e7da2af1ddb22c37eeeb7ae9b18dbc0a348fc87d41e47382daa2c8dd33b636 Oct 30 03:23:15.279097 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.279059 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" event={"ID":"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965","Type":"ContainerStarted","Data":"b8e4a20758a80460b999b2063d013744c1f6966c43a815d17dda7fdb5eb229d3"} Oct 30 03:23:15.279299 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.279131 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:15.279299 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.279159 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" event={"ID":"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965","Type":"ContainerStarted","Data":"829c4cc36df2e84fb8aa83acc95ecc463d75ee849fc0d9665ecd37e825777b16"} Oct 30 03:23:15.279299 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.279274 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:15.279464 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.279345 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" event={"ID":"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965","Type":"ContainerStarted","Data":"91e7da2af1ddb22c37eeeb7ae9b18dbc0a348fc87d41e47382daa2c8dd33b636"} Oct 30 03:23:15.280860 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.280840 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:15.320446 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:15.320378 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" podStartSLOduration=1.320361453 podStartE2EDuration="1.320361453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:15.319406302 +0000 UTC m=+1608.416822886" watchObservedRunningTime="2026-10-30 03:23:15.320361453 +0000 UTC m=+1608.417778012" Oct 30 03:23:22.292975 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:22.292948 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:23:24.328072 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.328043 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf"] Oct 30 03:23:24.331723 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.331702 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.346613 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.346589 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-82c29ebf98\"/\"isvc-logger-raw-cipn-kube-rbac-proxy-sar-config\"" Oct 30 03:23:24.346771 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.346700 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-gw0-82c29ebf98\"/\"odh-kserve-custom-ca-bundle\"" Oct 30 03:23:24.346854 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.346786 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.346933 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.346915 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.347009 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.346992 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.347082 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.347063 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x24tr\" (UniqueName: \"kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.347172 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.347155 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.356859 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.356837 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-82c29ebf98\"/\"seaweedfs-s3-creds\"" Oct 30 03:23:24.356969 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.356896 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-gw0-82c29ebf98\"/\"isvc-logger-raw-cipn-predictor-serving-cert\"" Oct 30 03:23:24.359799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.359780 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf"] Oct 30 03:23:24.448023 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.447992 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x24tr\" (UniqueName: \"kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448031 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448086 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448126 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448209 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448154 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448564 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448542 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448801 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448777 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.448923 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.448893 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.450632 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.450613 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.471095 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.471072 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x24tr\" (UniqueName: \"kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr\") pod \"isvc-logger-raw-cipn-predictor-86d746684c-zzwrf\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.651263 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.651176 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:24.973102 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:24.973032 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf"] Oct 30 03:23:24.976187 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:23:24.976159 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee566fa_f26b_46bf_83c3_46fe40f763b5.slice/crio-c79fab4c038a0f164540a37fe0b2ab17f8f3da3d9897c050b6cfdddae2edfaed WatchSource:0}: Error finding container c79fab4c038a0f164540a37fe0b2ab17f8f3da3d9897c050b6cfdddae2edfaed: Status 404 returned error can't find the container with id c79fab4c038a0f164540a37fe0b2ab17f8f3da3d9897c050b6cfdddae2edfaed Oct 30 03:23:25.320843 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:25.320735 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerStarted","Data":"c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251"} Oct 30 03:23:25.320843 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:25.320807 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerStarted","Data":"c79fab4c038a0f164540a37fe0b2ab17f8f3da3d9897c050b6cfdddae2edfaed"} Oct 30 03:23:29.336531 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:29.336495 5902 generic.go:356] "Generic (PLEG): container finished" podID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerID="c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251" exitCode=0 Oct 30 03:23:29.336970 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:29.336568 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerDied","Data":"c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251"} Oct 30 03:23:30.342577 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.342543 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerStarted","Data":"c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523"} Oct 30 03:23:30.342988 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.342612 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerStarted","Data":"79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316"} Oct 30 03:23:30.342988 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.342637 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerStarted","Data":"821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996"} Oct 30 03:23:30.342988 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.342939 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:30.343148 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.343006 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:30.344090 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.344064 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:23:30.407968 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:30.407924 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podStartSLOduration=2.780814316 podStartE2EDuration="6.407912389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.627098073s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:23:30.397497184 +0000 UTC m=+1623.494913744" watchObservedRunningTime="2026-10-30 03:23:30.407912389 +0000 UTC m=+1623.505328947" Oct 30 03:23:31.346621 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:31.346589 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:31.347116 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:31.346738 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:23:31.347878 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:31.347850 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:23:32.350312 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:32.350264 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:23:32.350710 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:32.350639 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:23:37.354583 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:37.354507 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:23:37.355065 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:37.355035 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:23:37.355339 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:37.355307 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:23:41.719038 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:23:41.719003 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:23:47.355006 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:47.354951 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:23:47.355504 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:47.355369 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:23:57.355083 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:57.355039 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:23:57.355569 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:23:57.355550 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:24:07.354942 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:07.354897 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:24:07.355424 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:07.355397 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:24:11.719641 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:24:11.719602 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:24:17.355695 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:17.355604 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:24:17.356387 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:17.356348 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:24:27.355177 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:27.355136 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:24:27.355698 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:27.355535 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:24:37.355579 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:37.355538 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:24:37.356143 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:37.356050 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:24:41.718605 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:24:41.718568 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:24:47.355417 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:47.355371 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:24:47.356067 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:47.355906 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:24:49.019277 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:49.019246 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:24:49.019742 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:49.019341 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:24:59.703185 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:24:59.703154 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/e2e-gw0-82c29ebf98_message-dumper-raw-cipn-predictor-7ff5658b48-s95ff_f2c0cf6f-bf94-47bb-8df7-7b2b6563f965/kserve-container/0.log" Oct 30 03:25:00.393451 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.393402 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf"] Oct 30 03:25:00.393957 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.393911 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" containerID="cri-o://821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996" gracePeriod=30 Oct 30 03:25:00.394121 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.393940 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" containerID="cri-o://c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523" gracePeriod=30 Oct 30 03:25:00.394121 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.393940 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" containerID="cri-o://79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316" gracePeriod=30 Oct 30 03:25:00.500590 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.500554 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff"] Oct 30 03:25:00.500951 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.500779 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kserve-container" containerID="cri-o://829c4cc36df2e84fb8aa83acc95ecc463d75ee849fc0d9665ecd37e825777b16" gracePeriod=30 Oct 30 03:25:00.500951 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.500862 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kube-rbac-proxy" containerID="cri-o://b8e4a20758a80460b999b2063d013744c1f6966c43a815d17dda7fdb5eb229d3" gracePeriod=30 Oct 30 03:25:00.672133 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.672106 5902 generic.go:356] "Generic (PLEG): container finished" podID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerID="b8e4a20758a80460b999b2063d013744c1f6966c43a815d17dda7fdb5eb229d3" exitCode=0 Oct 30 03:25:00.672133 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.672131 5902 generic.go:356] "Generic (PLEG): container finished" podID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerID="829c4cc36df2e84fb8aa83acc95ecc463d75ee849fc0d9665ecd37e825777b16" exitCode=2 Oct 30 03:25:00.672313 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.672182 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" event={"ID":"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965","Type":"ContainerDied","Data":"b8e4a20758a80460b999b2063d013744c1f6966c43a815d17dda7fdb5eb229d3"} Oct 30 03:25:00.672313 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.672245 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" event={"ID":"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965","Type":"ContainerDied","Data":"829c4cc36df2e84fb8aa83acc95ecc463d75ee849fc0d9665ecd37e825777b16"} Oct 30 03:25:00.674727 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.674696 5902 generic.go:356] "Generic (PLEG): container finished" podID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerID="79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316" exitCode=0 Oct 30 03:25:00.674853 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.674733 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerDied","Data":"79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316"} Oct 30 03:25:00.751301 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.751276 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:25:00.865443 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.865398 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config\") pod \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " Oct 30 03:25:00.865635 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.865507 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6\" (UniqueName: \"kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6\") pod \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " Oct 30 03:25:00.865635 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.865538 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls\") pod \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\" (UID: \"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965\") " Oct 30 03:25:00.865865 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.865811 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config" pod "f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" (UID: "f2c0cf6f-bf94-47bb-8df7-7b2b6563f965"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:00.867759 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.867740 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6" pod "f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" (UID: "f2c0cf6f-bf94-47bb-8df7-7b2b6563f965"). InnerVolumeSpecName "kube-api-access-5tmz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:00.867837 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.867763 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls" pod "f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" (UID: "f2c0cf6f-bf94-47bb-8df7-7b2b6563f965"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:25:00.967077 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.967038 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:00.967077 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.967078 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmz6\" (UniqueName: \"kubernetes.io/projected/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-kube-api-access-5tmz6\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:00.967262 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:00.967087 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:01.680220 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:01.680134 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" event={"ID":"f2c0cf6f-bf94-47bb-8df7-7b2b6563f965","Type":"ContainerDied","Data":"91e7da2af1ddb22c37eeeb7ae9b18dbc0a348fc87d41e47382daa2c8dd33b636"} Oct 30 03:25:01.680220 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:01.680166 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff" Oct 30 03:25:01.680220 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:01.680202 5902 scope.go:122] "RemoveContainer" containerID="b8e4a20758a80460b999b2063d013744c1f6966c43a815d17dda7fdb5eb229d3" Oct 30 03:25:01.688143 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:01.688127 5902 scope.go:122] "RemoveContainer" containerID="829c4cc36df2e84fb8aa83acc95ecc463d75ee849fc0d9665ecd37e825777b16" Oct 30 03:25:01.729174 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:01.729140 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff"] Oct 30 03:25:01.739695 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:01.739667 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-82c29ebf98/message-dumper-raw-cipn-predictor-7ff5658b48-s95ff"] Oct 30 03:25:02.350485 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:02.350447 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 30 03:25:03.019025 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:03.018994 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" path="/var/lib/kubelet/pods/f2c0cf6f-bf94-47bb-8df7-7b2b6563f965/volumes" Oct 30 03:25:05.698937 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:05.698907 5902 generic.go:356] "Generic (PLEG): container finished" podID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerID="821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996" exitCode=0 Oct 30 03:25:05.699338 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:05.698984 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerDied","Data":"821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996"} Oct 30 03:25:07.350853 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:07.350751 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 30 03:25:09.015417 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:09.015370 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:25:09.015883 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:09.015746 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:25:11.717920 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:25:11.717883 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:25:12.350976 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:12.350933 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 30 03:25:12.351149 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:12.351064 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:25:17.350429 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:17.350346 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 30 03:25:19.015177 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:19.015128 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:25:19.015672 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:19.015450 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:25:22.350556 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:22.350511 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 30 03:25:27.351395 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:27.351356 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused" Oct 30 03:25:29.014536 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:29.014478 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused" Oct 30 03:25:29.014997 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:29.014768 5902 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 03:25:29.017708 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:29.017689 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:25:29.017805 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:29.017795 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:25:30.547716 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.547688 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:25:30.588352 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588324 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert\") pod \"fee566fa-f26b-46bf-83c3-46fe40f763b5\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " Oct 30 03:25:30.588515 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588405 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls\") pod \"fee566fa-f26b-46bf-83c3-46fe40f763b5\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " Oct 30 03:25:30.588515 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588425 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config\") pod \"fee566fa-f26b-46bf-83c3-46fe40f763b5\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " Oct 30 03:25:30.588515 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588471 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location\") pod \"fee566fa-f26b-46bf-83c3-46fe40f763b5\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " Oct 30 03:25:30.588515 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588488 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr\" (UniqueName: \"kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr\") pod \"fee566fa-f26b-46bf-83c3-46fe40f763b5\" (UID: \"fee566fa-f26b-46bf-83c3-46fe40f763b5\") " Oct 30 03:25:30.588797 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588716 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert" pod "fee566fa-f26b-46bf-83c3-46fe40f763b5" (UID: "fee566fa-f26b-46bf-83c3-46fe40f763b5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:30.588881 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588808 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location" pod "fee566fa-f26b-46bf-83c3-46fe40f763b5" (UID: "fee566fa-f26b-46bf-83c3-46fe40f763b5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:25:30.588881 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.588859 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config" pod "fee566fa-f26b-46bf-83c3-46fe40f763b5" (UID: "fee566fa-f26b-46bf-83c3-46fe40f763b5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 03:25:30.590579 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.590553 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls" pod "fee566fa-f26b-46bf-83c3-46fe40f763b5" (UID: "fee566fa-f26b-46bf-83c3-46fe40f763b5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 03:25:30.590579 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.590555 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr" pod "fee566fa-f26b-46bf-83c3-46fe40f763b5" (UID: "fee566fa-f26b-46bf-83c3-46fe40f763b5"). InnerVolumeSpecName "kube-api-access-x24tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:25:30.689175 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.689143 5902 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fee566fa-f26b-46bf-83c3-46fe40f763b5-proxy-tls\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:30.689175 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.689177 5902 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:30.689370 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.689187 5902 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fee566fa-f26b-46bf-83c3-46fe40f763b5-kserve-provision-location\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:30.689370 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.689197 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x24tr\" (UniqueName: \"kubernetes.io/projected/fee566fa-f26b-46bf-83c3-46fe40f763b5-kube-api-access-x24tr\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:30.689370 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.689206 5902 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fee566fa-f26b-46bf-83c3-46fe40f763b5-cabundle-cert\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:25:30.791069 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.791037 5902 generic.go:356] "Generic (PLEG): container finished" podID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerID="c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523" exitCode=0 Oct 30 03:25:30.791250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.791105 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" Oct 30 03:25:30.791250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.791122 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerDied","Data":"c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523"} Oct 30 03:25:30.791250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.791232 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf" event={"ID":"fee566fa-f26b-46bf-83c3-46fe40f763b5","Type":"ContainerDied","Data":"c79fab4c038a0f164540a37fe0b2ab17f8f3da3d9897c050b6cfdddae2edfaed"} Oct 30 03:25:30.791373 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.791281 5902 scope.go:122] "RemoveContainer" containerID="c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523" Oct 30 03:25:30.799668 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.799651 5902 scope.go:122] "RemoveContainer" containerID="79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316" Oct 30 03:25:30.806688 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.806669 5902 scope.go:122] "RemoveContainer" containerID="821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996" Oct 30 03:25:30.814935 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.814909 5902 scope.go:122] "RemoveContainer" containerID="c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251" Oct 30 03:25:30.822566 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.822548 5902 scope.go:122] "RemoveContainer" containerID="c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523" Oct 30 03:25:30.822848 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:25:30.822811 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523\": container with ID starting with c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523 not found: ID does not exist" containerID="c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523" Oct 30 03:25:30.822925 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.822857 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523"} err="failed to get container status \"c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523\": rpc error: code = NotFound desc = could not find container \"c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523\": container with ID starting with c72651336141f40f1ecc33bab9f559402bd74b9c73ce5ad0f76d844df65cc523 not found: ID does not exist" Oct 30 03:25:30.822925 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.822877 5902 scope.go:122] "RemoveContainer" containerID="79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316" Oct 30 03:25:30.823151 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:25:30.823130 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316\": container with ID starting with 79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316 not found: ID does not exist" containerID="79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316" Oct 30 03:25:30.823250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.823153 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316"} err="failed to get container status \"79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316\": rpc error: code = NotFound desc = could not find container \"79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316\": container with ID starting with 79db1145750641688400d0497a389c08fc414171e121eaeb6d02c102d650e316 not found: ID does not exist" Oct 30 03:25:30.823250 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.823165 5902 scope.go:122] "RemoveContainer" containerID="821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996" Oct 30 03:25:30.823396 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:25:30.823379 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996\": container with ID starting with 821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996 not found: ID does not exist" containerID="821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996" Oct 30 03:25:30.823435 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.823401 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996"} err="failed to get container status \"821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996\": rpc error: code = NotFound desc = could not find container \"821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996\": container with ID starting with 821be616a563c8102e01a9bf919764ebd3ae62dc4adc661aeb4496e478913996 not found: ID does not exist" Oct 30 03:25:30.823435 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.823412 5902 scope.go:122] "RemoveContainer" containerID="c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251" Oct 30 03:25:30.823622 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:25:30.823605 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251\": container with ID starting with c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251 not found: ID does not exist" containerID="c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251" Oct 30 03:25:30.823664 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.823627 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251"} err="failed to get container status \"c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251\": rpc error: code = NotFound desc = could not find container \"c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251\": container with ID starting with c36fa7fb80773b905e408780611a28053260eb3ff66c59b3678ef0805851c251 not found: ID does not exist" Oct 30 03:25:30.845676 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.845648 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf"] Oct 30 03:25:30.850629 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:30.850611 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-gw0-82c29ebf98/isvc-logger-raw-cipn-predictor-86d746684c-zzwrf"] Oct 30 03:25:31.018507 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:31.018435 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" path="/var/lib/kubelet/pods/fee566fa-f26b-46bf-83c3-46fe40f763b5/volumes" Oct 30 03:25:33.249612 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249579 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rqq2b/must-gather-td7cm"] Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249865 5902 cpu_manager.go:414] "containerMap: removing container" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kserve-container" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249877 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kserve-container" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249887 5902 cpu_manager.go:414] "containerMap: removing container" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kube-rbac-proxy" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249892 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kube-rbac-proxy" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249909 5902 cpu_manager.go:414] "containerMap: removing container" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249914 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249926 5902 cpu_manager.go:414] "containerMap: removing container" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249930 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249937 5902 cpu_manager.go:414] "containerMap: removing container" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="storage-initializer" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249941 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="storage-initializer" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249954 5902 cpu_manager.go:414] "containerMap: removing container" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249958 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.249998 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kserve-container" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.250006 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kserve-container" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.250013 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2c0cf6f-bf94-47bb-8df7-7b2b6563f965" containerName="kube-rbac-proxy" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.250020 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="agent" Oct 30 03:25:33.250038 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.250026 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee566fa-f26b-46bf-83c3-46fe40f763b5" containerName="kube-rbac-proxy" Oct 30 03:25:33.254571 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.254553 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.273483 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.273457 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rqq2b/must-gather-td7cm"] Oct 30 03:25:33.282469 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.282434 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rqq2b\"/\"default-dockercfg-hjwl8\"" Oct 30 03:25:33.282469 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.282461 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rqq2b\"/\"kube-root-ca.crt\"" Oct 30 03:25:33.282641 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.282541 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rqq2b\"/\"openshift-service-ca.crt\"" Oct 30 03:25:33.310811 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.310779 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output\") pod \"must-gather-td7cm\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.310956 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.310834 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7ml\" (UniqueName: \"kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml\") pod \"must-gather-td7cm\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.411699 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.411663 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output\") pod \"must-gather-td7cm\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.411699 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.411701 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7ml\" (UniqueName: \"kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml\") pod \"must-gather-td7cm\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.412015 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.411996 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output\") pod \"must-gather-td7cm\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.435663 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.435634 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7ml\" (UniqueName: \"kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml\") pod \"must-gather-td7cm\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.583444 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.583367 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:25:33.889886 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.889859 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rqq2b/must-gather-td7cm"] Oct 30 03:25:33.891968 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:25:33.891940 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48df8efe_6de9_4bc3_8b48_6821500446d9.slice/crio-dc5b137e3035aaf5d4d10d984269205e76526ca2075ede626f5c96a77339937c WatchSource:0}: Error finding container dc5b137e3035aaf5d4d10d984269205e76526ca2075ede626f5c96a77339937c: Status 404 returned error can't find the container with id dc5b137e3035aaf5d4d10d984269205e76526ca2075ede626f5c96a77339937c Oct 30 03:25:33.893646 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:33.893627 5902 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 03:25:34.813200 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:34.813146 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rqq2b/must-gather-td7cm" event={"ID":"48df8efe-6de9-4bc3-8b48-6821500446d9","Type":"ContainerStarted","Data":"dc5b137e3035aaf5d4d10d984269205e76526ca2075ede626f5c96a77339937c"} Oct 30 03:25:40.840180 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:40.840147 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rqq2b/must-gather-td7cm" event={"ID":"48df8efe-6de9-4bc3-8b48-6821500446d9","Type":"ContainerStarted","Data":"f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8"} Oct 30 03:25:40.840564 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:40.840189 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rqq2b/must-gather-td7cm" event={"ID":"48df8efe-6de9-4bc3-8b48-6821500446d9","Type":"ContainerStarted","Data":"2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4"} Oct 30 03:25:40.880162 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:25:40.880112 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rqq2b/must-gather-td7cm" podStartSLOduration=1.653650364 podStartE2EDuration="7.880096902s" totalImagesPullingTime="6.226446538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 03:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:25:40.879941553 +0000 UTC m=+1753.977358125" watchObservedRunningTime="2026-10-30 03:25:40.880096902 +0000 UTC m=+1753.977513455" Oct 30 03:25:41.719256 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:25:41.719218 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:26:11.719188 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:26:11.719153 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:26:41.718945 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:26:41.718857 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:27:11.720583 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:27:11.720543 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:27:38.267058 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:38.267027 5902 generic.go:356] "Generic (PLEG): container finished" podID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerID="2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4" exitCode=0 Oct 30 03:27:38.267476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:38.267105 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rqq2b/must-gather-td7cm" event={"ID":"48df8efe-6de9-4bc3-8b48-6821500446d9","Type":"ContainerDied","Data":"2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4"} Oct 30 03:27:38.267476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:38.267453 5902 scope.go:122] "RemoveContainer" containerID="2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4" Oct 30 03:27:39.124190 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:39.124162 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rqq2b_must-gather-td7cm_48df8efe-6de9-4bc3-8b48-6821500446d9/gather/0.log" Oct 30 03:27:41.719610 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:27:41.719575 5902 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rflzz" containerName="extractor" Oct 30 03:27:44.667411 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:44.667375 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-569p6_feaaf2fd-3a0d-46a8-9252-9ea1909150df/global-pull-secret-syncer/0.log" Oct 30 03:27:44.857799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:44.857764 5902 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-rqq2b/must-gather-td7cm"] Oct 30 03:27:44.858006 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:44.857976 5902 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-must-gather-rqq2b/must-gather-td7cm" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="copy" containerID="cri-o://f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8" gracePeriod=2 Oct 30 03:27:44.869788 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:44.869759 5902 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-rqq2b/must-gather-td7cm"] Oct 30 03:27:44.871435 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:27:44.871404 5902 status_manager.go:1045] "Failed to get status for pod" err="pods \"must-gather-td7cm\" is forbidden: User \"system:node:ip-10-0-142-236.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-rqq2b\": no relationship found between node 'ip-10-0-142-236.ec2.internal' and this object" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:27:45.097355 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.097333 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rqq2b_must-gather-td7cm_48df8efe-6de9-4bc3-8b48-6821500446d9/copy/0.log" Oct 30 03:27:45.099793 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.099773 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:27:45.134084 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.134045 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml\" (UniqueName: \"kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml\") pod \"48df8efe-6de9-4bc3-8b48-6821500446d9\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " Oct 30 03:27:45.134243 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.134147 5902 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output\") pod \"48df8efe-6de9-4bc3-8b48-6821500446d9\" (UID: \"48df8efe-6de9-4bc3-8b48-6821500446d9\") " Oct 30 03:27:45.135518 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.135479 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output" pod "48df8efe-6de9-4bc3-8b48-6821500446d9" (UID: "48df8efe-6de9-4bc3-8b48-6821500446d9"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 03:27:45.136272 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.136251 5902 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml" pod "48df8efe-6de9-4bc3-8b48-6821500446d9" (UID: "48df8efe-6de9-4bc3-8b48-6821500446d9"). InnerVolumeSpecName "kube-api-access-jd7ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 03:27:45.235185 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.235108 5902 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/48df8efe-6de9-4bc3-8b48-6821500446d9-must-gather-output\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:27:45.235185 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.235135 5902 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd7ml\" (UniqueName: \"kubernetes.io/projected/48df8efe-6de9-4bc3-8b48-6821500446d9-kube-api-access-jd7ml\") on node \"ip-10-0-142-236.ec2.internal\" DevicePath \"\"" Oct 30 03:27:45.293014 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.292985 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-rqq2b_must-gather-td7cm_48df8efe-6de9-4bc3-8b48-6821500446d9/copy/0.log" Oct 30 03:27:45.293339 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.293312 5902 generic.go:356] "Generic (PLEG): container finished" podID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerID="f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8" exitCode=143 Oct 30 03:27:45.293511 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.293373 5902 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rqq2b/must-gather-td7cm" Oct 30 03:27:45.293511 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.293393 5902 scope.go:122] "RemoveContainer" containerID="f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8" Oct 30 03:27:45.301210 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.301190 5902 scope.go:122] "RemoveContainer" containerID="2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4" Oct 30 03:27:45.313479 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.313463 5902 scope.go:122] "RemoveContainer" containerID="f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8" Oct 30 03:27:45.313720 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:27:45.313705 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8\": container with ID starting with f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8 not found: ID does not exist" containerID="f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8" Oct 30 03:27:45.313769 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.313728 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8"} err="failed to get container status \"f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8\": rpc error: code = NotFound desc = could not find container \"f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8\": container with ID starting with f5a71665ede013cf4982581c926ed04a75f5009fd3dd962de08fc53da13614f8 not found: ID does not exist" Oct 30 03:27:45.313769 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.313744 5902 scope.go:122] "RemoveContainer" containerID="2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4" Oct 30 03:27:45.313980 ip-10-0-142-236 kubenswrapper[5902]: E1030 03:27:45.313964 5902 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4\": container with ID starting with 2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4 not found: ID does not exist" containerID="2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4" Oct 30 03:27:45.314061 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.313984 5902 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4"} err="failed to get container status \"2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4\": rpc error: code = NotFound desc = could not find container \"2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4\": container with ID starting with 2b228d7516bb16e123e24ae1f0c11303249789cf6b54fddd7b4d13db61b97ee4 not found: ID does not exist" Oct 30 03:27:45.957481 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:45.957449 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vhplh_9c32c037-b7a6-4a81-9ce9-478c53e023dd/konnectivity-agent/0.log" Oct 30 03:27:46.359286 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:46.359213 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-236.ec2.internal_0c527eada420e7f4995c8eee6de2ab03/haproxy/0.log" Oct 30 03:27:47.018464 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:47.018436 5902 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" path="/var/lib/kubelet/pods/48df8efe-6de9-4bc3-8b48-6821500446d9/volumes" Oct 30 03:27:50.844429 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:50.844381 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/alertmanager/0.log" Oct 30 03:27:50.996996 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:50.996964 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/config-reloader/0.log" Oct 30 03:27:51.124321 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:51.124249 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/kube-rbac-proxy-web/0.log" Oct 30 03:27:51.249068 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:51.249040 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/kube-rbac-proxy/0.log" Oct 30 03:27:51.374708 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:51.374636 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/kube-rbac-proxy-metric/0.log" Oct 30 03:27:51.497982 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:51.497956 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/prom-label-proxy/0.log" Oct 30 03:27:51.621190 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:51.621165 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_88ee36a1-9206-4764-8fd9-b6c164fcb996/init-config-reloader/0.log" Oct 30 03:27:51.889571 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:51.889541 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hq4tc_85d227a6-403a-4b90-9750-eb26433e0758/kube-state-metrics/0.log" Oct 30 03:27:52.015586 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:52.015559 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hq4tc_85d227a6-403a-4b90-9750-eb26433e0758/kube-rbac-proxy-main/0.log" Oct 30 03:27:52.139088 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:52.139061 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-hq4tc_85d227a6-403a-4b90-9750-eb26433e0758/kube-rbac-proxy-self/0.log" Oct 30 03:27:52.396490 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:52.396463 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-j69dd_eb7fca2c-37ba-41b9-b68e-897f5fb0e8d9/monitoring-plugin/0.log" Oct 30 03:27:52.917602 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:52.917576 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffn7c_d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/node-exporter/0.log" Oct 30 03:27:53.043143 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.043119 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffn7c_d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/kube-rbac-proxy/0.log" Oct 30 03:27:53.166918 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.166894 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ffn7c_d25c69c2-ac9b-492c-9332-bd1e3abd1f1b/init-textfile/0.log" Oct 30 03:27:53.497383 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497353 5902 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct"] Oct 30 03:27:53.497715 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497699 5902 cpu_manager.go:414] "containerMap: removing container" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="copy" Oct 30 03:27:53.497798 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497719 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="copy" Oct 30 03:27:53.497798 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497750 5902 cpu_manager.go:414] "containerMap: removing container" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="gather" Oct 30 03:27:53.497798 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497758 5902 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="gather" Oct 30 03:27:53.497981 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497853 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="copy" Oct 30 03:27:53.497981 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.497871 5902 memory_manager.go:365] "RemoveStaleState removing state" podUID="48df8efe-6de9-4bc3-8b48-6821500446d9" containerName="gather" Oct 30 03:27:53.500799 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.500778 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.516740 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.516717 5902 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8jkc8\"/\"default-dockercfg-ph9l8\"" Oct 30 03:27:53.528529 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.528501 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jkc8\"/\"openshift-service-ca.crt\"" Oct 30 03:27:53.528686 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.528532 5902 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8jkc8\"/\"kube-root-ca.crt\"" Oct 30 03:27:53.529257 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.529236 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct"] Oct 30 03:27:53.602788 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.602750 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-proc\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.602788 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.602786 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-podres\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.603035 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.602839 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-sys\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.603035 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.602931 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-lib-modules\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.603035 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.602970 5902 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f6hs\" (UniqueName: \"kubernetes.io/projected/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-kube-api-access-9f6hs\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704273 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704231 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-proc\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704273 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704268 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-podres\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704294 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-sys\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704342 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-lib-modules\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704358 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-proc\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704365 5902 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f6hs\" (UniqueName: \"kubernetes.io/projected/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-kube-api-access-9f6hs\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704401 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-sys\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704359 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-podres\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.704476 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.704448 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-lib-modules\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.730166 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.730140 5902 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f6hs\" (UniqueName: \"kubernetes.io/projected/c1a148be-afe4-4d68-8579-4ff9df5a5b1f-kube-api-access-9f6hs\") pod \"perf-node-gather-daemonset-l8pct\" (UID: \"c1a148be-afe4-4d68-8579-4ff9df5a5b1f\") " pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:53.813023 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:53.812937 5902 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:54.107421 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.107342 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/prometheus/0.log" Oct 30 03:27:54.109892 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.109868 5902 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct"] Oct 30 03:27:54.111914 ip-10-0-142-236 kubenswrapper[5902]: W1030 03:27:54.111889 5902 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc1a148be_afe4_4d68_8579_4ff9df5a5b1f.slice/crio-bb684a22f4fcd84e84500dba4514a67e8db563d47d4515e9a59c57fb1d1f3422 WatchSource:0}: Error finding container bb684a22f4fcd84e84500dba4514a67e8db563d47d4515e9a59c57fb1d1f3422: Status 404 returned error can't find the container with id bb684a22f4fcd84e84500dba4514a67e8db563d47d4515e9a59c57fb1d1f3422 Oct 30 03:27:54.230447 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.230390 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/config-reloader/0.log" Oct 30 03:27:54.326718 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.326683 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" event={"ID":"c1a148be-afe4-4d68-8579-4ff9df5a5b1f","Type":"ContainerStarted","Data":"86015bf3dfb8db7fb8b03902b6d52ca232baba25eb8732fd18395b2deb3136e6"} Oct 30 03:27:54.326897 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.326764 5902 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" event={"ID":"c1a148be-afe4-4d68-8579-4ff9df5a5b1f","Type":"ContainerStarted","Data":"bb684a22f4fcd84e84500dba4514a67e8db563d47d4515e9a59c57fb1d1f3422"} Oct 30 03:27:54.326897 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.326857 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:27:54.354198 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.354175 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/thanos-sidecar/0.log" Oct 30 03:27:54.368993 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.368920 5902 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" podStartSLOduration=1.368907648 podStartE2EDuration="1.368907648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 03:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 03:27:54.36795851 +0000 UTC m=+1887.465375068" watchObservedRunningTime="2026-10-30 03:27:54.368907648 +0000 UTC m=+1887.466324204" Oct 30 03:27:54.478027 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.478002 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/kube-rbac-proxy-web/0.log" Oct 30 03:27:54.603623 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.603600 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/kube-rbac-proxy/0.log" Oct 30 03:27:54.729772 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.729744 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/kube-rbac-proxy-thanos/0.log" Oct 30 03:27:54.853312 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.853282 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0beb25d6-9d3f-48e7-a846-36680fa37e88/init-config-reloader/0.log" Oct 30 03:27:54.989920 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:54.989841 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-lzxzh_4259b75f-532b-4e73-9d4e-0d3d72909feb/prometheus-operator/0.log" Oct 30 03:27:55.111857 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:55.111813 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-lzxzh_4259b75f-532b-4e73-9d4e-0d3d72909feb/kube-rbac-proxy/0.log" Oct 30 03:27:55.376749 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:55.376679 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cbbcbd748-fpx9f_62a522bc-31b4-477f-8a93-4d8959c0aabf/telemeter-client/0.log" Oct 30 03:27:55.499558 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:55.499500 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cbbcbd748-fpx9f_62a522bc-31b4-477f-8a93-4d8959c0aabf/reload/0.log" Oct 30 03:27:55.622838 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:27:55.622795 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cbbcbd748-fpx9f_62a522bc-31b4-477f-8a93-4d8959c0aabf/kube-rbac-proxy/0.log" Oct 30 03:28:00.340611 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:00.340584 5902 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8jkc8/perf-node-gather-daemonset-l8pct" Oct 30 03:28:03.505404 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:03.505375 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gp6sv_0ab8f2d0-573d-495b-9075-bced70a5f27b/dns/0.log" Oct 30 03:28:03.646019 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:03.645994 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gp6sv_0ab8f2d0-573d-495b-9075-bced70a5f27b/kube-rbac-proxy/0.log" Oct 30 03:28:04.197805 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:04.197773 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-pkc7w_02ef4e11-7e6a-49ad-a017-d51735f3b124/dns-node-resolver/0.log" Oct 30 03:28:05.721281 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:05.721245 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fsp8w_23217ed3-9c56-47de-be14-b34a31cc29e2/node-ca/0.log" Oct 30 03:28:08.749605 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:08.749530 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hpxr7_c663f5b1-25fa-4b8a-8dc7-95f273fecd53/serve-healthcheck-canary/0.log" Oct 30 03:28:10.681899 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:10.681871 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rflzz_0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/kube-rbac-proxy/0.log" Oct 30 03:28:10.804975 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:10.804948 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rflzz_0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/exporter/0.log" Oct 30 03:28:10.929329 ip-10-0-142-236 kubenswrapper[5902]: I1030 03:28:10.929298 5902 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rflzz_0ef3e93e-b3dc-42ed-942e-8b808a9dfeeb/extractor/0.log"